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Tactical  Decision  Aids  and  Situational  Awareness 

(RTO  EN-019  /  SCI-113) 


Executive  Summary 

This  Report  documents  the  results  of  NATO  Research  and  Technology  Organization  (RTO)  SCI-113 
Lecture  Series  number  LS  227,  entitled  “Tactical  Decision  Aids  and  Situational  Awareness”. 

This  Lecture  Series  has  been  sponsored  by  the  Systems  Concepts  and  Integration  (SCI)  Panel  and 
the  material  contained  in  this  publication  was  presented  on  1-2  November,  2001  in  Amsterdam, 
The  Netherlands,  on  8-9  November,  2001  in  Sofia,  Bulgaria,  on  12-13  November,  2001  in  Madrid, 
Spain  and  on  19-20  November,  2001  at  the  Patuxent  River  Naval  Air  Station,  Maryland,  USA. 

The  primary  purpose  of  this  Lecture  Series  was  to  focus  the  LS  audience  on  the  current  scientific  and 
technical  knowledge  within  the  domain  of  Decision  Aids  Systems  in  relation  to  certain  ongoing 
development  programs. 

The  authors  of  the  Lecture  Series  covered  in  particular  the  major  problems  to  be  addressed  in  the 
requirements  definition,  the  state-of-the-art,  the  emerging  technologies,  the  achievements,  the  expected 
benefits  to  the  end-users,  the  lessons  learned  and  the  future  trends. 

Due  to  the  fact  that  in  the  complex  and  fast-paced  Battlespace  of  the  future,  humans  will  rely  more  and 
more  on  Information  Technology  to  deliver  knowledge  and  to  assist  them  in  using  that  knowledge,  the 
decisions  will  be  reached  by  a  mix  of  human  and  machine  reasoning. 

The  aim  of  the  Decision  Aids  Systems  is  to  achieve  the  decide  and  act  capability. 

The  key  enabling  technologies  to  provide  such  a  capability,  as  described  in  the  Lecture  Series,  can  be 
found  in  the  area  of  the  Information  Technology  and  in  the  automation  process  of  the  man-machine 
integration,  together  with  the  accurate  modelling  of  the  human  cognitive  processes. 

Special  emphasis  was  given  during  the  Lecture  Series  to  the  description  of  programs  covering: 

•  Interaction  of  human  perception  and  judgement  with  automated  information  processing  and 
presentation 

•  Mission  Management  and  Crew  Assistance  for  Military  Aircraft 

•  Pilot  oriented  workload  evaluation  and  redistribution 

•  Interacting  Multiple  Model  Approach  in  Dynamic  Situation 

The  material  in  this  publication  was  assembled  to  support  a  Lecture  Series  under  the  sponsorship  of  the 
Systems  Concepts  and  Integration  (SCI)  Panel  and  the  Consultant  and  Exchange  Programme  of  RTO 
presented  on  1-2  November  2001  in  Amsterdam,  The  Netherlands,  on  8-9  November  2001  in  Sofia, 
Bulgaria,  on  12-13  November  2001  in  Madrid,  Spain  and  on  19-20  November  2001  in  Maryland,  USA. 
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Les  aides  a  la  prise  de  decisions  tactiques  et 
la  connaissance  de  la  situation  des  forces 

(RTO  EN-019  /  SCI-113) 

Synthese 

Ce  rapport  presente  les  resultats  du  Cycle  de  conferences  LS  227  sur  “Les  aides  a  la  prise  de  decisions 
tactiques  et  la  connaissance  de  la  situation  des  forces”  organise  par  la  Commission  sur  les  concepts  et 
1’  integration  de  systemes  (SCI-113)  de  P  Organisation  pour  la  recherche  et  la  technologie  de  LOT  AN 
(RTO). 

Dans  le  cadre  de  cette  activite,  les  textes  contenus  dans  cette  publication  ont  ete  presentes  du  1  au  2 
novembre  2001  a  Amsterdam,  Pays-Bas,  du  8  au  9  novembre  2001  a  Sofia,  Bulgarie,  du  12  au  13 
novembre  2001  a  Madrid  en  Espagne  et  du  19  au  20  novembre  2001  a  la  base  aeronavale  de  Patuxent 
River,  Maryland  aux  Etats-Unis. 

Ce  cycle  de  conferences  a  eu  pour  objectif  principal  de  presenter  Tetat  actuel  des  connaissances 
scientifiques  et  techniques  dans  le  domaine  des  systemes  d’aides  a  la  prise  de  decisions,  tel  que  reflete 
par  un  certain  nombre  de  programmes  de  developpement  actuels. 

Les  conferenciers  ont  notamment  developpe  les  principaux  problemes  a  aborder  dans  le  cadre  de  la 
definition  des  specifications,  Tetat  actuel  des  connaissances,  les  technologies  naissantes,  les 
realisations,  les  benefices  escomptes  pour  Tutilisateur  final,  les  enseignements  tires  et  les  tendances 
futures. 

Etant  donne  que  les  acteurs  du  champ  de  bataille  complexe  et  dynamique  du  futur  feront  appel  de  plus 
en  plus  a  des  technologies  de  V information  pour  transmettre  les  connaissances  et  pour  etre  aide  dans 
leur  exploitation,  les  decisions  seront  prises  par  le  biais  d’un  processus  decisionnel  homme-machine. 

Le  but  des  systemes  d’aide  a  la  prise  de  decisions  est  de  parvenir  a  une  capacite  du  type  “decider  et 
agir”. 

Comme  il  est  expose  dans  le  cycle  de  conferences,  les  technologies  cles  permettant  de  fournir  une  telle 
capacite  se  trouvent  dans  le  domaine  des  technologies  de  Tinformation,  dans  le  processus 
d’automatisation  de  T integration  homme-machine  ainsi  que  dans  la  modelisation  precise  des  processus 
cognitifs  humains. 

Une  attention  particuliere  a  ete  portee  a  la  description  de  programmes  couvrant  : 

•  L’interaction  entre  le  jugement  et  la  perception  de  Thomme  et  la  presentation  et  le  traitement  de 
Tinformation  automatisee. 

•  La  gestion  de  la  mission  et  Taide  aux  equipages  des  aeronefs  militaires. 

•  L’evaluation  et  la  redistribution  de  la  charge  de  travail  des  pilotes. 

•  L’approche  du  modele  interactif  multiple  en  situation  dynamique. 

Cette  publication  a  ete  redigee  pour  servir  de  support  de  cours  pour  le  cycle  de  conferences  organise 
par  la  commission  sur  les  concepts  et  Tintegration  de  systemes  (SCI)  )  du  1  au  2  novembre  2001  a 
Amsterdam  aux  Pays  Bas,  du  8  au  9  novembre  2001  a  Sofia  en  Bulgarie,  du  12  au  13  novembre  2001  a 
Madrid  en  Espagne  et  du  19-20  novembre  2001  a  Patuxent  River  aux  Etats-Unis. 
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Introduction  -  Technical  Overview  and  State  of  the  Art 

Dr.  Ing.  Luigi  Crovelia 

Societa’  Italiana  Avionica-  SIA-  S.p.A. 

Strada  antica  di  Collegno  253 
10146  Torino,  Italy 
crovella@sia-av.it 


Objective 

Today  the  use  of  Decision  Aids  Systems  for  Commander  and  Operators  in  the  Battlefield  area  is 
playing  an  important  role  due  to  the  new  frequent  situation  of  joint  coalition  and  asymmetic 
warfare  in  which  Defense  Forces  are  involved. 

On  these  occasions,  the  capability  of  own  Forces  to  follow  the  evolution  of  the  Tactical  Situation 
in  real  time  is  extremely  important. 

Since  Combat  Survival  and  Mission  Accomplishment  depend  upon  Operators  performance  in  the 
process  of  decision-making,  and  the  Operators  performance  depends  upon  the  degree  of 
awareness,  Situation  Awareness  can  be  seen  as  a  result  of  a  continuous  assessment  of  situation 
parameters  by  the  Operators. 

This  Mission  critical  chain  of  sub-segment  functions  is  greatly  influenced  by  the  nature  of  the 
technical  systems  the  Operator  is  having  to  deal  with. 

The  purpose  of  the  present  Lecture  Series  is  to  provide  to  the  audience: 

•  Definition  of  the  problem 

•  Overview  of  the  state-of-the-art 

•  Description  of  some  research  and  development  programs 

•  Exposure  to  the  end-users  of  the  potential  benefits  of  the  decision  aids  employment 

•  Future  trends 


What  is  Decision? 

Decision ,  following  the  Webster’s  Dictionary,  means  “the  act  or  process  of  deciding ”,  and 
to  decide  means  “to  arrive  at  a  solution  that  ends  uncertainty  ”,  as  well  as  “to  make  a  choice 
This  definition  is  correct  for  our  purpose. 


What  is  a  Decision  Aid? 

In  today’s  information-intensive  Battlefield,  Operators  need  decision  aids  to  free  them  from 
information  overload. 

Decision  aiding  technologies  fuse  data  from  onboard  sensors  and  outside  sources  to: 

•  Create  a  composite  picture  of  the  battlefield 

•  Recognize  potential  threats 

•  Impart  key  information  to  friend  forces. 

•  Instantaneously  update  mission  plans. 


Paper  presented  at  the  RTO  SCI  Lecture  Series  on  (<Tactical  Decision  Aids  and  Situational  Awareness  ”,  held  in  Amsterdam, 
The  Netherlands,  1-2  November  2001;  Sofia,  Bulgaria,  8-9  November  2001;  Madrid,  Spain,  12-13  November  2001; 
Maryland,  United  States,  19-20  November  2001,  and  published  in  RTO-EN-0 19. 
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Key  Elements  of  Decision- Aiding  Software  are  typically: 

•  Data  fusion  -  combines  track  information  from  a  variety  of  sources  into  a  single  best 
picture  of  the  Battlefield 

•  Situation  assessment  -  continually  monitors  this  dynamic  picture  for  impacts  to  the  plan 

•  Mission  planners  -  recommend  updates  to  the  plan 

•  Execution  aids  -  help  the  crew  in  executing  the  mission. 


What  is  Tactical  Decision  Making? 

In  line  with  the  above  definition,  tactical  decision-making  is  the  main  task  and  responsibility  of 
the  tactical  decision-maker ,  who  is  active  at  any  level  of  the  tactical  decision-making  process . 
Unfortunely,  the  decision  has  to  be  based  upon  tactical  military  information  and  operational 
environment  which  are,  by  definition,  always  uncertain  to  varying  degrees. 

Therefore  effective  tactical  decision-making  refers  primarely  to  the  ability  to  use  logical  and 
sound  judgement  to  make  decisions  on  available  information . 

The  available  information  is  developing  from  the  elementary  external  data  collected  by  sensor 
systems  to  the  situational  awareness  through  a  judgement  process. 

What  above  need  to  be  supported  by  appropriate  decision  aids  in  order  to  improve  operator 
performance. 

The  automation  of  the  process  has  to  be  coherent  with  the  above. 


What  is  Situational  Awareness? 

Situational  awareness  is  generally  defined  as  the  degree  of  accuracy  by  which  the  Operator’s 
perception  of  the  external  environment  reflects  the  reality. 

To  our  purpose,  situational  awareness  is  instead  to  be  defined  as  “the  ability  to  reliably , 
accurately  and  continuously  collect  information  on  the  situation ,  enemy  or  friendly ,  when  and 
where  required 

In  simple  words,  in  the  military  environment  “it  is  the  mechanism  which  pinpoints  targets  and 
threats  to  represent  the  Battlespace  situation  ”. 

It  is  always  to  be  reminded  that,  in  line  with  the  basic  definition,  awareness  is  a  matter  of  degree 
and  not  an  absolute. 


Situation  Awareness 

Situation  awareness  is  the  major  element  of  the  information  superiority  which  is  needed  for  the 
Battlespace  dominance  by  the  Systems  of  Systems ,  in  accordance  with  the  updated  Defense 
requirements. 


Information  Superiority 

Information  superiority  is  the  result  of  the  capability  to  gather ,  process ,  integrate ,  disseminate 
and  display  situation  awareness  information ,  together  with  a  corresponding  increase  in  the 
ability  to  use  that  information . 

This  ability  is  the  knowledge  of  the  Battlespace ,  which  is  necessary  to  make  decisions  and  take 
actions  allowing  to  dominate  the  Battlespace. 

A  very  careful  attention  has  to  be  put  in  obtaining  the  knowledge ,  so  that  it  is  coherent  with  the 
cognitive  processes. 
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Only  if  this  is  reached,  the  decision-making  is  optimized  and  the  man-machine  loop  is  closed 
according  to  the  requirements. 


Automation  aspects 

Automation  is  a  fundamental  issue  to  achieve  an  improved  operational  effectiveness  of  the 
decision-making  process. 

This  is  due  to  the  large  amounts  of  data  to  be  handled  in  dynamic  and  heavy  scenario,  which 
could  overwelm  Operators  capabilities. 

The  fundamental  problem  encountered  during  the  implementation  of  an  effective  automation  is 
how  support  is  provided  by  decision  aid  on  elements  of  the  decision-making. 

The  dualism  is  to  use  the  decision  aid  as  a  prosthesis  adding  additional  capabilities  to  the 
Operators  or  simply  as  a  tool  available  to  the  Operators;  the  differences  are  related  to  the  role  of 
the  aid  in  the  decision  process. 

The  prosthetic  approach  is  targeted  to  replace  the  Operators  when  the  situation  causes  an 
excessive  workload  that  cannot  be  managed  by  the  human  capabilities  and  to  provide  the  needed 
decision  outcomes. 

The  tool  approach  is  targeted  to  assist  the  Operators  active  role  in  accordance  with  the  decision- 
making  process  requirements. 

These  approaches  are  not  mutually  exclusive  but  complementary,  depending  on  situation  context, 
the  specific  nature  of  the  decision  aid  element  and  the  Operators  role. 

The  correct  choice  is  let  to  the  designers  capability  and  experience. 


Lecture  Series  Overview 

The  present  Lecture  Series  concentrates  on  the  discussion  of  examples  of  applications  covering  in 
particular: 

•  Interaction  of  human  perception  and  judgement  with  automated  information  processing 
and  presentation 

•  Mission  Management  and  Crew  assistance 

•  Approach  to  cognitive  and  cooperative  Operators  assistance  in  the  field  of  Tactical  Flight 
Mission  Management 

•  On-board  decision  support  techniques 

•  Pilot  oriented  workload  evaluation  and  redistribution 

•  Multiple  hypotheses  multiple  model  approach  techniques 

The  goal  is  that  the  detailed  presentations  together  with  the  discussions  of  achievements, 
problems  and  lessons  learned  from  the  Programs  shall  help  Decision  Aids  Systems  potential 
designers  and  users  in  defining  and  evaluating  operational  requirements  and  affordable  solutions. 

An  extensive  use  of  the  modem  design  and  development  processes  and  techniques  based  on  the 
Systems  Engineering,  Systems  Analysis,  Functional  Analysis,  Simulation  and  Rapid  Prototyping 
have  to  be  considered  for  the  success  of  the  Programs. 
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I.  Situational  Awareness  and  Understanding 


Dr.  Howard  S.  Marsh,  Mr.  Paul  W.  Quinn, 
Mr.  Gary  J.  Toth,  LCDR  David  A.  Jakubek 

Office  of  Naval  Research 
800  N.  Quincy  Street 
Arlington,  VA  22217-5660,  USA 


Tactical  decisions  are  made  under  conditions  best  described  as  “the  fog  and  friction  of  war”.  They  tend  to  be 
judgmental  rather  than  analytical  and  are  based  on  the  decision  maker’s  perception  of  the  situation  and  of  his 
or  her  options  for  meeting  objectives  defined  by  the  commander.  The  perception  and  the  resulting  decisions 
are  very  sensitive  to  the  quality  and  completeness  of  the  knowledge  that  the  decision  maker  obtains  through 
interactions  with  the  decision  support  systems. 

Modem  information  technology  provides  enormous  potential  for  expanded  situational  awareness  using  a 
variety  of  information  management,  display,  and  human-system  interaction  tools  that  can  help  the  decision 
makers  penetrate  the  “fog  of  war”  and  deal  with  the  “friction  of  war”.  On  the  other  hand,  the  increased  use  of 
automation  also  tends  to  remove  the  decision  makers  from  direct  observation  of  the  situation  and  requires  them 
to  rely  on  information  that  is  derived  or  inferred  by  processes  that  are  embedded  within  a  complex  system  of 
systems.  This  can  impede  the  j  udgmental  decision  process  due  to  lack  of  confidence  in  the  information  or  due 
to  a  desire  to  obtain  more  information  before  committing  to  a  course  of  action.  The  relationship  between  the 
human  judgmental  processes  and  the  automated  decision  support  systems  is  particularly  important  for  tactical 
combat  direction  and  execution,  where  the  pressure  to  decide  and  act  is  intense  and  where  the  results  of 
decisions  are  often  lethal. 

This  lecture  explores  the  process  in  which  the  decision  maker  achieves  an  awareness  and  understanding  of  the 
situation  based  on  observations  and  on  his  or  her  model*  of  the  world  and  of  the  current  operational  context. 


*A  model  of  the  world  is  simply  a  set  of  rules  and  relationships  that  describe  how  entities  and  events  occur  and  move 
through  time  and  space.  For  example,  our  world  model  tells  us  that  entities  that  move  through  the  air  are  either  birds  or 
aircraft  and  that  automobiles  do  not  move  through  the  air.  A  model  of  the  context  is  a  similar  set  of  rules  and 
relationships  that  describe  how  the  elements  of  the  world  model  are  related  to  the  current  operational  situation.  For 
example,  if  our  context  is  open  ocean  operations,  our  context  model  tells  us  that  entities  moving  along  the  surface  of  the 
earth  are  most  likely  ships  and  not  trains  or  automobiles.  Similarly,  if  the  context  is  peace,  then  our  context  model  tells  us 
that  incoming  aircraft  are  not  likely  to  be  attacking  us. 


Paper  presented  at  the  RTO  SCI  Lecture  Series  on  (<Tactical  Decision  Aids  and  Situational  Awareness  ”,  held  in  Amsterdam, 
The  Netherlands,  1-2  November  2001;  Sofia,  Bulgaria,  8-9  November  2001;  Madrid,  Spain,  12-13  November  2001; 
Maryland,  United  States,  19-20  November  2001,  and  published  in  RTO-EN-0 19. 
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A  Traditional  Model  for  Decision  Making 


Figure  1.  A  Traditional  Model  for  Decision  Making 


This  is  the  first  part  of  a  two-part  lecture  dealing  with  situational  awareness  and  tactical  decision  aids  and  the 
role  of  information  technology  in  supporting  the  operational  processes. 

Figure  1  above  shows  a  traditional  model  for  moving  from  observed  data  to  a  perception  of  the  situation  and 
then  to  a  decision  on  how  to  respond.  This  first  lecture  examines  the  cognitive  process  that  leads  to  perception 
and  understanding  of  the  situation  and  the  way  that  technology  is  used  to  support  that  process.  The  second 
lecture  examines  the  further  processes  for  reaching  a  decision  to  act  upon  that  understanding  of  the  situation. 

The  first  step  in  the  process  that  leads  to  a  decision  is  observation  of  the  real  world  through  all  available 
measurements,  human  inputs,  and  encyclopedic  information  stored  in  computers  and  in  documents.  These 
basic  fragments  of  information  are  then  expressed  as  digital  representations  called  data  so  that  they  can  be 
moved  and  managed  easily.  This  is  shown  in  Figure  2. 

A  very  important  factor  is  introduced  at  the  time  when  we  make  a  measurement  and  express  it  as  data.  The 
measurement  itself  is  an  observation  of  some  phenomenon  or  multiple  phenomena  associated  with  a  specific 
event  or  entity.  The  event  or  entity  is  fully  deterministic,  it  is  a  specific  and  precise  element  in  the  overall  state 
of  the  environment.  However,  the  act  of  observing  phenomena  associated  with  that  entity  or  event  produces 
data  that  contains  some  element  of  statistical  uncertainty.  Even  a  human  observation  of  an  event  or  entity  is  a 
probabilistic  determination  of  the  state  of  the  environment.  The  probability,  or  confidence  level,  of  the 
observation  may  be  very  high,  but  we  still  need  to  recognize  that  the  resulting  data  do  not  represent  “ground 
truth”. 
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Data 


•  A  Representation  of  Individual  Facts,  Concepts,  or  Instructions  in  a  Manner 
Suitable  for  Communication,  Interpretation,  or  Processing  by  Humans  or  by 
Automatic  Means.  [IEEE  Definition  Included  in  DOD  TRM] 

•  Data  Are  Numerical  Representations  of  Measured  Phenomena  [E.G.  Sensor 
Reports]  or  Inputs  Generated  by  Humans  or  Computers  to  Represent  Some 
Quantifiable  Aspect  of  an  Entity,  Event,  or  Environment. 


Observable  Space 
Precision  Limitations 
Time  Constraints 


Measurable 
Phenomena 

Human  and 
Computer  Inputs 


Data  Production 


Sensors 
Staffs 
Computers 
Data  Formats 
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Formatted  Numerical  Output 

For  Example: 

•  Radar  Returns 

•  Lines  of  Bearing 

•  Digitized  Imagery 

•  etc. 


Figure  2.  The  Process  Model  for  Producing  Data 

Thus,  the  initial  inputs  to  the  process  are  never  fully  deterministic.  There  is  always  some  degree  of 
uncertainty,  ambiguity,  and  credibility  associated  with  the  data  as  it  moves  through  the  system.  Bear  this  in 
mind  as  we  progress  through  the  stages  leading  to  a  perception  and  understanding  of  the  situation. 

The  difference  between  data  and  information  is  very  subtle  but  important.  Data  are  numerical  representations 
that  can  represent  measurements  or  other  inputs  from  human  beings  or  automated  systems.  Data  are  the 
vehicle  for  conveying  information  either  as  unassociated  fragments  or  as  thoroughly  processed  and  integrated 
products.  If  the  data  can  be  associated  with  a  specific  sensor  or  source,  the  meaning  of  that  data  becomes 
clear,  and  this  transforms  the  numerical  data  into  information  that  can  be  viewed  and  interpreted  by  human 
beings  or  automated  systems.  This  corresponds  to  correlating  the  numerical  data  with  some  model  of  the 
sensor  or  source  and  the  context  to  which  the  numbers  refer.  That  is  the  basic  essence  of  information. 

For  example,  if  a  radar  signal  is  reflected  from  an  aircraft,  the  observation  produces  numerical  data  related  to 
that  observation.  A  model  of  the  radar’s  functionality  tells  us  that  the  data  define  distance  from  the  radar,  size 
of  the  radar  cross  section  that  is  indicated  by  the  strength  of  the  return,  and  other  observables  such  as  velocity 
component  toward  or  away  from  the  radar  (for  Doppler  radars),  and  altitude  (for  height  finding  radars).  When 
multiple,  sequential  radar  returns  are  observed,  the  data  can  begin  to  include  the  full  velocity  vector.  The  radar 
data  now  is  interpreted  as  information:  a  target  of  a  certain  size,  speed,  and  location,  moving  through  the 
airspace. 

The  process  model  for  producing  information  is  illustrated  in  Figure  3. 
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Information 

•  The  Refinement  of  Data  Through  Known  Conventions  and  Context  for 
Purposes  of  Imparting  Knowledge.  [DOD  TRM] 

•  Information  Is  Produced  When  Data  Is  Expressed  Within  a  Context  That  Gives 
It  Meaning  in  Terms  of  the  Nature  of  the  Observed  or  Inferred  Entity,  Event, 
or  Environmental  Feature,  Including  Criteria  Associated  With  the  Producing 
System  Such  As  Precision  and  Confidence  Level. 

-  Contextual  or  Semantic  Interpretation 

-  Combination  of  Individual  Data  Associated  With  a  Common  Entity,  Event,  or 
Environmental  State  (e.g.  multiple  lines  of  bearing  or  multiple  radar  hits) 
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•  etc. 


Figure  3.  The  Process  Model  for  Producing  Information 

The  next  step  in  the  process  from  observing  to  understanding  is  to  convert  the  information  to  knowledge  of  the 
situation.  This  is  the  process  that  generates  a  perception  of  the  situation. 

Knowledge  is  produced  when  information  is  correlated  with  a  model  of  the  world  and  the  current  context. 
Consider  the  example  of  the  radar  return.  If  the  information  indicates  an  entity  at  an  altitude  of  10,000  feet 
and  a  speed  of  500  nautical  miles  per  hour,  the  “world  model”  tells  us  that  it  is  most  likely  a  fixed  wing 
aircraft,  and  most  likely  a  turbojet  powered  aircraft. 

As  more  information  is  correlated  with  this  radar  return,  we  may  know  even  more  about  it.  For  example,  IFF 
returns  may  indicate  that  it  is  a  friendly  military  aircraft,  or  other  types  of  information  may  indicate  that  it  is  a 
hostile  military  aircraft.  Each  of  these  additional  pieces  of  information  can  be  correlated  with  one  another, 
according  to  the  current  operational  context. 

Knowledge  often  builds  from  multiple  possible  interpretations,  called  multiple  hypotheses.  As  we  gather  more 
information,  we  decide  which  of  these  interpretations  to  believe,  and  this  produces  our  perception  of  the 
situation.  We  also  recognize  where  additional  information  is  needed,  and  this  produces  a  directed  search  for 
additional  information. 


This  process  is  illustrated  in  Figure  4. 


Building  Knowledge  from  Information: 
Perception  of  the  Situation 


Directed  Search  for  Information 


Figure  4,  Building  Knowledge  and  a  Perception  of  the  Situation 

The  production  of  knowledge  from  information  is  just  one  more  step  in  correlating  observations  with  our 
models  of  the  world  and  the  operational  context.  The  principal  factor  introduced  at  this  point  in  the  process  is 
the  need  to  infer  knowledge  based  on  an  interpretation  of  information  and  patterns  of  information  within  the 
framework  of  the  models  of  the  world  and  the  context.  This  is  where  the  human  cognitive  processes  become 
important  and  where  the  partnership  between  the  machines  and  the  humans  begins  to  become  complex. 

Inference  and  interpretation  are  based  to  a  great  extent  on  the  credibility  assigned  to  information.  This  is 
especially  true  when  we  have  multiple  hypotheses  to  consider  and  when  the  information  and  the  world  model 
and  context  model  do  not  align  perfectly  with  one  another.  Sometimes  we  can  identify  a  key  piece  of 
information  that  will  clearly  differentiate  among  the  hypotheses,  but  more  often  we  can  only  develop  our 
perception  with  an  imperfect  degree  of  confidence. 

Any  imprecision,  ambiguity,  or  uncertainty  in  the  source  data  will  carry  through  the  process  and  lead  to 
corresponding  imperfections  in  our  knowledge  and  our  perception  of  the  situation.  This  theme,  dealing  with 
uncertainties  and  ambiguities,  is  one  that  lies  at  the  heart  of  the  decision  support  automation  that  we  strive  to 
perfect.  Figure  5  shows  a  process  model  for  producing  knowledge. 
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Knowledge 

•  Knowledge  Is  the  Interpretation  of  Information  Concerning  Entities,  Events, 
and  Environmental  States  That  Results  in  a  Specific  Conclusion  About  the 
Physical  World  Based  on  the  Available  Information  and  Current  Context. 

•  Knowledge  Is  the  “As  Is”  Perception  of  the  Current  Situation  Based  on  a 
Contextual  Interpretation  and  Integration  of  Available  Information. 

•  Knowledge  Can  Be  Derived  From  Recognition  of  Patterns  of  Information 
Within  the  Current  Semantic  and  Contextual  Frames  of  Reference 

-  Correlation  of  Observables:  E.G.  HULTEC 

-  Correlation  of  Observed  Entities:  E.G.  Aggregation  of  Units  to  Organizations 

-  Interpretation  Based  on  Physical  Constraints:  E.G.  Inference  of  Entity  Type 


*•  Perception  of  the  Situation 
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•  etc. 
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Figure  5.  The  Process  Model  for  Producing  Knowledge 

We  now  introduce  the  notion  of  a  “perception  map”  as  shown  in  the  diagram  in  Figure  6.  We  draw  inferences 
from  this  base  of  knowledge,  assumptions,  and  recognized  uncertainties  using  judgment  and  experience. 
Human  beings  are  very  adept  at  correlating  current  patterns  of  information  with  past  experience,  especially 
when  information  is  displayed  as  visual  patterns  that  can  be  compared  with  previous  experiences. 

The  perception  map  is  a  useful  tool  in  understanding  why  we  reach  certain  conclusions,  why  we  are  sometimes 
surprised  by  outcomes  that  are  different  from  what  we  expected,  and  how  we  cope  with  uncertainties.  In 
general,  we  strive  to  move  from  the  “known  unknown”  sector  to  the  “known”  sector,  to  decrease  the  degree  of 
uncertainty. 

We  make  assumptions  when  we  cannot  resolve  “known  unknowns”  that  are  essential  to  the  perception.  In 
those  cases,  we  rely  on  judgment,  experience,  and  intuition  to  define  the  assumption.  The  danger  here  is  that 
we  may  forget  that  we  based  our  perception  on  a  critical  assumption  rather  than  a  “known”  set  of  information. 
We  may  also  forget  that  a  “known”  set  of  information  was  actually  based  on  imperfect  data,  as  noted  earlier. 
Consequently,  even  those  things  that  we  “know”  are  known  only  to  some  degree  of  confidence. 
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The  “perception  map”  shows  some  essential  features  of  the  perception  that  leads  to  our  understanding  of  the 
situation.  The  main  factor  is  the  inherently  statistical  nature  of  all  the  underlying  information.  We  can  know 
the  situation  only  as  well  as  we  can  make  observations  and  interpret  them.  If  observations  are  in  error,  we  will 
have  error  in  perceiving  the  situation.  If  models  of  the  world  or  the  operational  context  are  in  error,  we  will 
infer  incorrect  knowledge  from  the  information.  The  previous  example  of  inferring  that  an  approaching 
aircraft  is  not  attacking  was  based  on  an  operational  context  of  peace.  A  disastrous  error  in  perception  occurs 
when  we  think  we  are  at  peace  but  the  adversary  has  actually  made  a  transition  to  war  and  is  launching  a 
surprise  attack. 

While  the  “assumptions”,  the  “known  unknowns”,  and  the  imperfect  “knowns”  are  certainly  problems  to  be 
managed,  the  real  villains  in  the  perception  map  are  the  “unknown  unknowns”.  Ideally,  our  information  and 
our  models  of  the  world  and  context  are  good  enough  for  us  to  identify  all  the  critical  “unknowns”  and  move 
them  from  the  “unknown  unknown”  sector  to  the  “known  unknown”  sector.  Automated  information  systems 
can  be  powerful  tools  to  help  us  do  this,  since  they  can  track  vast  amounts  of  information  and  continually 
correlate  and  update  relationships  of  information  with  world  models  and  context  models. 

As  noted  in  Figure  7,  the  mere  fact  that  we  know  something  does  not  mean  that  we  understand  what  that 
knowledge  means.  We  may  know  too  little  about  a  situation  to  make  sense  of  it;  we  may  be  confronted  with 
“facts”  that  appear  to  contradict  one  another  and  that  tend  to  confuse  us;  or  we  may  know  too  much  and  be 
distracted  by  interesting  knowledge  at  the  expense  of  focusing  on  the  really  important  knowledge. 
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Understanding  Is  the  Basis  for  the  Decision 

“I  Don’t  Understand  Everything  I  Know.” 

•  This  Statement  Provides  the  Best  Insight  Into  the  Difference  Between 
Knowledge  and  Understanding 

•  Information  Systems  Need  to  Help  the  Decision-Maker  Understand  the 
Situation,  Not  Just  Know  It 

-  Situation  Awareness  Is  Not  the  Objective 

-  Situation  Understanding  Is  Our  Goal 

•  If  I  Know  Too  Much,  It  May  Impede  My  Understanding  of  the  Critical 
Information  —  Information  Overload,  “Glare  of  War” 

•  If  I  Know  Too  Little,  My  Predisposition  to  Assume  Things  May 
Distort  My  Perception  of  Reality 

-  Experts  Have  Greater  Predisposition  to  Assume  Than  Novices 

-  The  Judgment  and  Wisdom  That  Comes  With  Expertise  Also  Brings 
Intellectual  “Baggage” 


Figure  7.  The  Difference  Between  Knowledge  and  Understanding 

Another  factor  in  reaching  an  understanding  of  a  situation  is  the  tendency  of  human  beings  to  approach 
understanding  from  a  predisposition  to  assume  something  about  the  situation  even  before  any  information  is 
presented.  Prejudgment  is  a  fact  that  we  need  to  recognize.  When  we  have  little  information  and  little 
knowledge  of  the  situation,  most  of  our  perception  lies  in  the  “assumed”  and  “known  unknown”  sectors.  Our 
“knowledge”  or  awareness  of  the  situation  is  driven  by  the  assumptions  and  the  recognition  of  gaps  in 
knowledge.  This  can  tend  to  cause  our  prejudgment  to  dominate. 

The  process  for  moving  from  knowledge  to  understanding  relies  very  heavily  on  the  judgment  and  experience 
of  the  human  being  and  on  his  or  her  ability  to  correlate  past  experiences,  training,  and  education  with  the 
current  context. 

Constraints  on  understanding  the  situation  include  gaps  in  knowledge,  preconceived  notions  of  what  the 
situation  should  be,  and  time  pressures.  The  fog  and  friction  of  war  are  clearly  at  play  at  this  point. 

Since  judgment  is  a  major  factor  in  understanding  the  situation,  the  inputs  to  the  process  certainly  include 
credibility  factors  related  to  the  information  that  has  been  presented.  Here  we  probably  find  a  “feedback  loop” 
back  to  the  process  that  converted  information  to  knowledge.  If  we  think  that  we  know  something  but  cannot 
understand  it,  we  will  try  to  “rethink”  the  process  from  observed  information  to  interpretation  in  terms  of 
knowledge  or  perception  of  the  situation.  This  is  the  typical  question:  “What’s  wrong  with  this  picture?” 

Figure  8  illustrates  a  process  model  for  developing  understanding  from  knowledge  of  the  situation. 
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Understanding 

•  Understanding  Is  the  Interpretation  of  Knowledge  in  Terms  of  Meaning 
Within  the  Current  Operational  Context. 

-  Recognition  of  the  Meaning  of  Recognized  Patterns  in  the  Perceived  Situation 

-  Interpretation  of  the  “As  Is”  Knowledge  in  Terms  of  the  Dynamics  of  the  Situation 

-  Identification  of  Uncertainties,  Ambiguities,  and  Needs  for  More  Information 

-  Identification  of  Requirements  to  Respond  to  the  Situation 

•  Understanding  Applies  Judgment  to  Interpret  Knowledge  in  Terms  of  Past 
Experience,  Training,  and  Other  Factors  That  Allow  Patterns  to  Be  Associated 
With  Both  the  Static  and  Dynamic  Aspects  of  the  Observed  Situation. 
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Figure  8.  The  Process  Model  for  Building  Understanding 

Understanding  is  achieved  through  a  correlation  of  the  perception  with  the  likely  alternatives  (multiple 
hypotheses)  of  what  that  perception  means  in  terms  of  the  current  operational  context.  That  correlation  is  then 
reviewed  in  terms  of  the  context  to  produce  an  understanding  of  what  it  means.  Note  that  the  context  model 
plays  an  important  role  in  moving  from  information  to  knowledge  (perception)  and  then  to  understanding.  We 
continually  “fine  tune”  our  view  of  the  information  by  matching  our  j  udgments,  conclusions,  and  assumptions 
with  the  context  model.  That  model  itself  grows  and  becomes  more  precise  and  specific  as  we  use  the  process 
to  update  the  model.  As  we  know  more  about  the  situation,  we  adjust  the  context  model,  and  we  use  that 
adjusted  model  to  make  further  progress  toward  understanding.  This  is  illustrated  in  Figure  9. 

We  also  can  fool  ourselves  by  making  errors  in  assuming  or  adjusting  a  context  model  based  upon  conclusions 
that  were  reached  by  using  the  context  model  itself.  If  we  believe  strongly  enough  that  a  certain  situation 
exists,  we  may  find  evidence  to  reinforce  that  belief  even  when  it  is  false.  In  engineering  parlance,  if  we  build 
an  amplifier  with  enough  gain  and  with  fine  enough  tuning,  we  will  see  the  signal  we  seek  even  if  it  is  not 
there;  we  will  have  produced  an  oscillator  instead  of  an  amplifier.  A  basic  tenet  in  deception  tactics  is  to  show 
something  that  is  false  but  that  is  expected,  thereby  reinforcing  a  predisposition  to  reach  wrong  conclusions 
about  the  situation. 
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Figure  9.  The  Correlation  and  Feedback  Processes  that  Build  Understanding 

To  this  point  we  have  described  the  situational  awareness  and  understanding  process  as  one  in  which  we 
correlate  and  integrate  information  to  build  a  picture,  or  perception,  of  the  situation.  In  a  very  simple  situation, 
where  information  is  clear,  precise,  and  credible,  that  linear  process  of  integrating  and  correlating  may  work, 
but  in  the  real  world,  we  often  need  to  look  into  the  details  of  the  information  or  we  need  to  acquire  more 
information  in  order  to  reach  conclusions.  This  is  shown  as  the  directed  search  for  information  in  Figure  9. 
The  search  can  be  within  the  files  of  information  that  are  already  available,  or  it  can  be  in  the  form  of  tasking 
for  collection  of  additional  information  to  resolve  “known  unknowns”  or  to  confirm  assumptions. 

The  need  for  examining  specific  details,  also  called  “drilldown”,  arises  when  those  details  are  essential  for 
knowing  how  much  credibility  to  assign  to  information  or  when  we  find  that  we  may  have  to  adjust  our 
context  model  based  on  evidence.  We  often  want  to  examine  the  evidence  in  detail  before  making  that  kind  of 
adjustment.  Drilldown  is  also  needed  when  the  automated  processes  or  formal  rules  and  algorithms  for 
building  knowledge  produce  results  that  are  either  unexpected,  ambiguous,  or  confusing.  By  seeing  specific 
pieces  of  information,  we  can  often  use  human  judgment  and  experience  to  do  a  better  job  of  sorting  through 
the  information  than  the  automated  processes  or  the  predefined  rules  and  algorithms. 

We  now  recognize  that  the  “perception  map”  is  not  as  simple  as  it  first  appeared.  Recall  that  all  the  source 
data  was  statistical  in  nature.  That  is  the  unfortunate  result  of  having  to  rely  on  imperfect  measurements  of 
phenomena  to  determine  the  state  of  the  real  world  as  noted  earlier  in  this  lecture.  Consequently,  even  when 
we  think  that  we  know  something,  we  only  know  it  to  a  limited  degree  of  accuracy  and  confidence. 

Accuracy  is  something  that  we  hope  to  have  defined  for  us,  based  on  the  specific  sensors  and  processors  that 
deliver  the  information. 
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Figure  10.  The  “Perception  Map”  With  Additional  Details  Related  to  Confidence  and  Importance 

Confidence  and  importance  are  often  difficult  to  determine  and  may  be  influenced  by  many  factors,  including 
the  “strength”  of  the  observed  phenomenon,  the  experience  of  human  beings  who  may  have  interpreted  the 
observations,  alternative  hypotheses  associated  with  the  observations,  known  unknowns,  and  so  forth. 

The  diagram  in  Figure  10  is  not  intended  to  be  a  precise  description  of  how  confidence  and  importance  affect 
the  interaction  among  known,  assumed,  and  unknown  aspects  of  the  perception.  It  is  only  a  notional  diagram 
to  help  display  how  those  factors  affect  our  overall  knowledge  and  understanding  of  the  situation. 

Understanding  of  a  situation  is,  in  many  respects,  like  putting  together  the  pieces  of  a  puzzle.  When  we  have 
all  the  pieces,  when  the  fully  assembled  picture  is  one  that  is  simple,  and  when  each  piece  has  parts  of  that 
picture  that  define  where  it  fits,  we  can  construct  the  full  picture.  However,  when  some  pieces  are  missing  or 
when  we  find  pieces  from  other  puzzles  that  look  similar  to  one  another  (alternate  hypotheses),  we  have 
difficulty  constructing  the  picture.  This  is  the  challenge  in  understanding  a  complex  situation  with  numerous 
alternative  interpretations  and  with  information  that  is  statistical  in  nature  and  that  has  many  missing  pieces. 

When  confronted  with  such  a  challenge,  we  need  to  search  for  the  information  that  is  most  critical  to  making 
the  judgment  and  eliminating  the  unlikely  alternative  interpretations.  This  is  called  “value  of  information”. 
We  look  for  the  missing  pieces  that  will  have  the  greatest  influence  on  our  determination  rather  than  for  less 
influential  ones.  We  also  look  among  the  ambiguous  pieces  of  information  for  the  ones  that  make  the  greatest 
difference  in  our  conclusions.  This  process  of  “diagnosing”  the  situation  is  identical  to  the  process  that  a 
physician  uses  to  diagnose  a  disease.  This  is  illustrated  in  Figure  11. 
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Mathematical  Rigor  for  Dealing  With 
Relationships  Among  Uncertain  Information 


Figure  1 1.  A  Bayesian  Network  Formalism  for  Understanding  a  Set  of  Observations 

As  previously  noted,  the  “diagnosis”  of  the  situation  rests  on  key  elements  of  information  and  on  relationships 
among  them.  Those  relationships  can  be  expressed  as  a  network  of  nodes  and  links  that  show  way  that 
observed  information  can  be  traced  to  specific  inferences  about  conditions  or  states  of  the  observed  system  that 
can  be  made  from  the  observations. 

In  this  example,  condition  8  at  the  lower  right  hand  comer  of  the  network  is  distinguished  from  condition 
8  only  by  observation  F.  All  other  observations  are  consistent  with  both  of  them,  even  though  the  likelihood 
of  the  two  “hypotheses”  (i.e.  condition  <|>  or  condition  8)  may  be  different  based  on  those  observations. 
Furthermore,  if  condition  8  does  exist,  then  we  can  infer  that  conditions  a,  (3,  and  y  also  exist,  since  condition  8 
depends  on  observations  that  imply  those  other  conditions  too. 

The  question  of  likelihood  is  important  in  making  the  diagnosis.  Bayesian  nets  allow  the  appearance  of  an 
observation  or  condition  to  be  expressed  in  terms  other  than  100%  or  0%.  We  can  then  produce  a  diagnosis 
that  gives  probability  estimates  for  each  conclusion  based  on  the  probabilities  assigned  to  the  source 
information.  In  the  example  of  the  radar  return  from  an  aircraft,  the  probabilities  might  be  expressed  as:  (fixed 
wing  100%),  (friendly  80%,  hostile  5%,  unknown  15%),  (attacking  2%,  not  attacking  90%,  unknown  8%). 

The  diagnostic  net  in  Figure  1 1  showed  that  observation  F  was  the  most  important  piece  of  information  to 
distinguish  between  conditions  8  and  Therefore,  if  we  need  to  make  that  distinction,  we  need  to  try  to 
observe  F. 
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The  tactical  forces  deal  with  similar  demands  for  distinguishing  between  various  states  of  the  situation.  Is  the 
enemy  over  that  hill  or  has  he  moved  to  some  other  location?  Am  I  about  to  be  attacked?  These  questions 
demand  focused  observation  to  collect  the  most  valuable  pieces  of  information.  It  could  be  information  that 
fills  a  void  or  it  could  be  information  to  resolve  ambiguous  conclusions  from  information  that  we  already  have. 

Sensors  to  collect  such  information  are  an  important  part  of  the  situational  awareness  resources.  They  serve  to 
provide  many  of  the  initial  inputs  during  the  Intelligence  Preparation  of  the  Battlespace  (IPB)  and  through 
ongoing  surveillance  and  reconnaissance,  and  they  also  serve  to  respond  to  immediate  needs  for  information 
based  on  questions,  issues,  and  problems  that  arise  as  we  process  information  to  obtain  knowledge  and  then  to 
understand  that  knowledge.  Technology  exists  to  build  sensors  that  are  suited  to  real  time  tasking  and 
management  by  tactical  forces,  and  this  will  help  them  penetrate  the  “fog  of  war”  in  a  way  that  is  responsive  to 
their  fine-grained,  and  locally  focused  needs. 


Once  information  is  collected  and  processed  to  provide  a  basis  for  understanding,  we  need  to  address  the 
human  cognitive  aspects  of  the  process.  This  means  that  we  have  to  organize  and  present  the  information  so 
that  the  human  beings  can  use  it  effectively.  Unlike  computers,  human  beings  do  not  simply  manipulate 
numbers  according  to  predefined  mathematical  rules.  They  are  more  adept  at  recognizing  patterns  of 
information  and  comparing  them  with  past  experience  or  training.  Consequently,  the  way  that  information  is 
presented  needs  to  focus  on  displaying  those  patterns  explicitly  and  without  requiring  the  user  to  waste  time 
and  effort  in  peripheral  tasks,  such  as  extracting  information  from  unformatted  text.  Figure  12  is  notional, 
intended  to  make  the  point  that  pictures  are  better  than  words  most  of  the  time  and  that  formatted  presentations 
are  easier  to  work  with  than  simple  narrative  text. 


Impact  of  Presentation  Form 


Delivery  Rate 


Figure  12.  The  Form  of  Presentation  Impacts  the  Ability  of  Humans  to  Use  Information 
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Modem  command  and  control  systems  are  built  with  these  facts  in  mind.  Most  information  is  presented  as 
some  form  of  graphic  display:  a  map,  a  bar  chart,  and  so  forth.  This  provides  the  overarching  view  and 
pattern;  it  establishes  context  and  a  framework  for  interpreting  information;  and  it  guides  the  users  to  areas 
where  they  may  want  to  “drill  down”  to  see  more  information.  The  next  level,  as  they  drill  down,  is  often  a 
brief,  formatted  display  of  amplifying  data.  Then,  if  they  need  more,  they  often  can  retrieve  textual 
information  or  source  data.  The  principle  is  first  to  present  the  overview,  often  general  and  qualitative,  and  to 
let  the  users  determine  how  much  they  need  in  terms  of  actual  numerical  data  or  textual  references. 


Impact  of  the  System  Interaction  Method 

•  The  Best  Human-system  Interface  Combines  the 
Interactions  With  the  Normal  Process  of  Doing  the  Task 
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•  WIMP  -  Windows,  Icons,  Mouse,  Point-and-Click  Is  Better 

•  Embedding  the  Interaction  Mechanism  in  the  Task  Material 
(E.G.  An  Active  Situation  Map  or  Display)  Is  Even  Better 

•  Modem  Systems  Attempt  to  Offer  a  Human-system 
Interface  That  Is  a  “Metaphor”  of  the  Workspace 
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Figure  13.  Implications  of  Various  Forms  of  Human-System  Interface 

The  summary  in  Figure  13  is  based  on  experiences  with  real  world  experience  and  is  manifestly  evident  to 
anyone  who  has  worked  with  automated  systems.  Any  requirement  to  use  human  intellect  to  punch  keys  or 
read  narrative  text  is  wasteful.  Efficiency  is  gained  as  the  human-system  interface  becomes  more  oriented  on 
the  environment  in  which  information  is  appraised  and  manipulated.  If  the  workspace  is  structured  as  a  map 
with  overlays  and  icons,  the  WIMP  interface  works  well.  If  the  icons  can  be  constructed  with  summary 
information  displayed  explicitly  and  drilldown  information  embedded  and  easily  accessed  with  point-and- 
click,  the  interface  becomes  extremely  useful. 

Other  features  can  be  added  to  the  human-system  interface  to  reduce  the  burden  of  accessing  and  manipulating 
information.  Some  of  these  features  can  be  built  as  “intelligent”  software  agents  that  can  be  tasked  to  do  some 
of  the  routine  and  burdensome  tasks.  For  example,  an  agent  can  be  used  to  track  specific  types  of  information 
and  alert  the  human  operator  when  certain  predefined  conditions  are  observed.  Other,  more  complex  agents 
can  infer  need  to  alert  the  human  operator  when  specific  patterns  of  information  are  observed  or  when 
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ambiguities  and  contradictions  are  observed.  Speech  interface  is  also  a  feature  of  some  new  systems,  currently 
in  the  commercial  world  but  making  their  way  into  military  command  and  control  systems.  We  can  expect 
significant  advances  that  begin  to  provide  actual  human-computer  dialog  and  multi-agent  cooperative  tasking. 


Figure  14  shows  an  integrated  information  and  multi -agent  command  and  control  tool  that  was  built  by  Logica 
UK  for  a  U.S.  Marine  Corps  and  Navy  Advanced  Concept  Technology  Demonstration  (ACTD)  called 
“Extending  the  Littoral  Battlespace”  (ELB).  The  Logica  system  was  an  adaptation  of  work  of  a  European 
consortium  led  by  Logica.  The  display  and  human-system  interface  embody  many  of  the  principles  discussed 
previously. 


Figure  14.  An  Implementation  of  Integrated  Information  Services 


The  workspace  metaphor  is  a  map.  This  is  a  very  common  preference  for  operational  and  tactical  command 
and  control  functions.  Logica  provides  overlays  for  various  types  of  information,  such  as  tactical  situation, 
meteorological  effects,  and  terrain  reasoning.  The  overlays  are  managed  by  software  agents,  and  each  overlay 
can  be  turned  on  or  off,  depending  on  the  amount  of  information  that  the  human  operator  wants  to  see  at  the 
time.  This  provides  user  control  over  the  degree  of  integrated  information  that  is  displayed.  Some  of  the 
agents  are  very  simple  ones  that  just  place  icons  or  other  information  on  the  overlay.  Others  are  more  complex 
and  do  tasks  such  as  terrain  reasoning,  line-of-sight  calculation,  and  so  forth.  The  Logica  interface  and  the 
underlying  agent-oriented  environment  provided  a  very  powerful  tool  for  helping  the  users  work  with  a 
network  of  computers  that  contained  a  very  broad  and  complex  set  of  information.  By  using  the  power  of  the 
interface  and  its  tools,  the  user  could  construct  precisely  the  display  that  suited  his  or  her  task. 
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Figure  15.  A  Representative  Display  of  Tactical  Situation  Data 

The  Office  of  Naval  Research  is  managing  a  program  called  Dominant  Battlespace  Command  (DBC).  It  is 
being  executed  by  Concurrent  Technologies  Corporation,  with  the  objective  to  build  visualization 
environments  that  interface  with  currently  available  command  and  control  information  systems  to  improve  a 
commander’s  ability  to  comprehend  the  battlespace  situation  and  make  decisions  quickly  and  effectively. 

This  DBC  display  shown  in  Figure  1 5  is  connected  to  the  track  data  base  management  (TDBM)  software  in  the 
Global  Command  and  Control  System  (GCCS).  The  interface  allows  the  users  to  see  both  the  overall  tactical 
situation  and  selected  details  (drilldown)  of  specific  information  for  each  entity  in  the  track  data  base.  This 
type  of  drilldown  capability  is  a  standard  feature  in  GCCS  and  many  other  command  and  control  systems  that 
are  currently  in  use.  It  is  an  important  way  to  present  the  overall  pattern  without  saturating  the  user  with  too 
much  detail,  and  still  allow  the  user  to  obtain  those  details  when  needed. 

The  map  display  is  really  a  “rich  meta  data”  display  rather  than  a  display  of  actual  data.  It  is  like  a  merger  of 
the  “index”  and  “annotated  outline”  of  the  content  in  the  data  base.  It  provides  an  overall  sense  of  the  situation 
and  the  relationships  among  entities  and  events  that  have  been  observed.  When  we  need  the  actual  data,  it 
provides  pointers  to  the  selected  information. 

Users  of  the  track  display  may  want  to  see  specific  entities  and  not  others.  This  is  especially  important  when 
the  display  is  cluttered  with  objects  that  are  not  of  immediate  interest.  Filters  provide  a  capability  to  select  the 
types  of  entities  that  are  shown  and  the  ones  that  are  maintained  in  the  background.  This  is  another  important 
capability  provided  by  systems  such  as  GCCS. 
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When  we  use  filters  to  declutter  a  display,  we  need  to  be  careful  to  assure  that  the  hidden  entities  are  called  to 
our  attention  if  necessary.  Otherwise,  we  can  reach  conclusions  about  the  overall  situation  and  take  actions 
that  may  be  in  conflict  with  the  actual  state  of  the  battlespace.  This  can  cause  unintentional  “blue  on  blue”  or 
“blue  on  white”  engagements  or  dangerous  conflicts  of  fires  and  maneuver.  This  is  where  software  agents  can 
be  established  to  monitor  the  hidden  entities  and  alert  the  users  when  certain  criteria  are  met. 

The  DBC  display  environment  and  many  other  command  and  control  displays  also  allow  users  to  select  the 
specific  view  of  the  situation  that  they  need  at  the  time.  In  the  case  illustrated  in  Figure  16,  the  user  has 
selected  the  tactical  tracks  from  the  GCCS  track  data  base  overlaid  on  a  two  dimensional  map. 
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Figure  16.  Selection  of  Data  Sources  to  Be  Displayed 

Systems  such  as  this  present  the  users  with  an  active  display  of  their  choice,  an  ability  to  drilldown  for  detailed 
information,  and  a  list  of  other  available  displays  that  are  reached  with  a  simple  point-and-click. 
Consequently,  the  users  have  available  to  them  a  very  rich  source  of  situational  information,  with  an  ability  to 
structure  the  view  of  that  information  to  suit  their  needs. 

Another  important  visualization  tool  is  the  explicit  display  of  surveillance  and  tracking  coverage  areas  or 
volumes.  This  allows  us  to  recognize  regions  where  unknown  information  may  reside.  It  helps  convert 
“unknown  unknowns”  to  “known  unknowns”  so  that  we  can  deal  with  them.  The  display  in  Figure  1 7  focuses 
on  the  coverage  of  a  U2  reconnaissance  aircraft  as  it  flies  through  the  battlespace.  The  “fan”  extending  from 
the  U2  toward  the  coast  shows  precisely  where  the  U2  sensors  are  looking  at  the  time.  This  allows  the 
operators  to  know  what  areas  are  being  covered  and  what  areas  are  not  covered. 


Figure  17.  Visualization  of  the  Sensor  Coverage  of  a  U2  Surveillance  Aircraft 

The  information  in  this  display  can  also  be  presented  as  a  three  dimensional  perspective  display  if  that  suits  the 
users.  Other  options  for  the  presentation  include  a  true  3-D  display  or  an  immersive  reality  environment.  The 
DBC  program  is  experimenting  with  all  of  these  types  of  user  interfaces  to  determine  how  each  one  could 
contribute  to  situation  awareness  and  understanding  for  different  types  of  tasks  and  operational  situations. 

We  conclude  the  first  part  of  the  two-part  lecture  with  the  summary  in  Figure  18. 

Situational  awareness  must  always  contend  with  “fog  of  war”.  As  we  are  able  to  observe,  process,  and 
disseminate  more  information  about  the  battlespace,  some  of  the  fog  will  be  removed,  but  we  can  never  see 
true  “ground  truth”.  Uncertainties  and  ambiguities  will  always  remain  and  must  be  tracked  and  managed.  In 
fact,  as  we  bring  more  sensors  to  bear  and  connect  more  users  with  one  another,  the  potential  for  ambiguities 
and  contradictions  increases.  We  see  this  in  some  of  the  current  tactical  data  link  systems  where  the  number  of 
unresolved  duplications  of  track  and  target  data  increases  as  we  add  more  sensors.  Consequently,  we  have 
greater  need  for  automated  tools  and  models  to  resolve  ambiguities  and  contradictions. 

The  second  problem  area  that  we  need  to  address  is  what  can  be  called  the  “glare  of  war”.  This  is  the 
overabundance  of  information  that  is  delivered  to  the  users  and  that  can  obscure  the  information  that  they 
actually  need.  The  fact  that  information  will  now  come  in  many  different  forms  and  at  different  rates 
compounds  the  problem  of  presenting  an  integrated,  coherent,  and  relatively  simple  picture  to  the  human 
beings  so  that  they  can  recognize  the  important  patterns  and  not  be  distracted  by  irrelevant  information.  We 
need  to  make  sure  that  additional  information  helps  their  performance. 
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Information-Leveraged  Warfare  Presents  New 
Challenges  as  Well  as  Opportunities 

•  The  “Fog  of  War”:  A  Continuing  Challenge 

-  Uncertainty:  Let’s  Not  Fool  Ourselves  -  We  Are  Not  Omniscient! 

-  Ambiguity:  More  Information  May  Increase  Ambiguity 

-  Assumptions:  Forever  Necessary  to  Fill  Critical  Information  Gaps 

-  Unknown  Unknowns 

•  The  “Glare  of  War”:  The  New  Challenge 

-  Massive  Inputs  of  Information 

-  Mix  of  Relevant  and  Marginally  Useful  Information 

-  Heterogeneous  Forms  of  Information  and  Presentation 
—  Hiding  of  Important  Information  Within  the  “Clutter” 

Too  Much  Information  Can  Be  As  Bad  As  Too  Little. 
The  Key  Is  In  The  Management  And  Presentation. 


Figure  18.  Summary:  Challenges  and  Opportunities  in  Information-Leveraged  Warfare 
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The  Interaction  of  Human  Perception  and  Judgment  with  Automated 
Information  Processing  and  Presentation 

II.  Decision  Support 
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Modem  information  technology  can  improve  situational  awareness  and  understanding  far  beyond  the 
traditional  “fog  of  war”,  but  these  improvements  are  useful  only  if  the  operators  can  apply  that  awareness  and 
understanding  to  reach  decisions  better  and  faster  than  before.  The  goal  is  to  achieve  “decision  superiority” 
not  j  ust  information  superiority. 

In  the  complex  and  fast-paced  battlespace  of  the  future,  humans  will  rely  more  and  more  on  information 
technology  to  deliver  knowledge  and  to  assist  them  in  using  that  knowledge.  As  the  range,  speed,  lethality, 
and  cost  of  weapons  increase,  the  human  cognitive  processes  will  have  to  be  augmented  by  rapid,  automated 
appraisal  of  the  situation  and  the  available  courses  of  action.  The  decisions  will  have  to  be  reached  by  a  mix 
of  human  and  machine  reasoning,  including  a  theater-wide  perspective  as  well  as  a  local  perspective.  This 
presents  new  challenges  for  the  system  developers  and  the  operational  concept  developers,  and  those 
challenges  are  being  addressed  in  a  number  of  development  and  experimentation  programs. 

This  discussion  extends  the  prior  consideration  of  situational  awareness  and  understanding  to  address  the  use 
of  that  awareness  and  understanding  to  determine  the  decision. 


Paper  presented  at  the  RTO  SCI  Lecture  Series  on  (<Tactical  Decision  Aids  and  Situational  Awareness  ”,  held  in  Amsterdam, 
The  Netherlands,  1-2  November  2001;  Sofia,  Bulgaria,  8-9  November  2001;  Madrid,  Spain,  12-13  November  2001; 
Maryland,  United  States,  19-20  November  2001,  and  published  in  RTO-EN-0 19. 
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Three  Types  of  Decision  Drivers 

•  Procedure 

-  Decisions  That  Are  Made  According  to  a  Schedule 

-  Examples:  the  Daily  Briefing  to  the  Boss,  the  ATO,  Ordering  Lunch 

•  Need 

-  Event-driven 

-  A  Decision  Must  Be  Made  to  Respond  to  the  Event 

-  Examples:  Destroy  Incoming  Missile,  Steer  Car  Around  Pothole 

•  Opportunity 

-  Event-driven 

-  If  a  Decision  Can  Be  Made  Quickly  Enough,  Some  Unanticipated 
Gain  May  Result 

-  Example:  Enemy  CP  Sighted,  One-day  Price  Reduction 


Figure  1 .  Reasons  for  Making  Decisions 
We  make  decisions  for  one  of  three  reasons  as  indicated  in  Figure  1 . 

An  important  goal  of  modem  knowledge  superiority  or  decision  superiority  is  to  allow  us  to  move  from 
procedural  decision  making  toward  opportunistic  decision  making.  This  allows  us  to  dominate  the  battlespace 
and  to  take  the  initiative  away  from  our  adversary.  We  still  need  to  be  able  to  respond  to  events  beyond  our 
control,  but  we  strive  to  minimize  this  type  of  reactive  posture  and  move  as  much  as  possible  toward  a 
proactive  posture. 

Procedural  decision  making,  on  the  other  hand,  is  largely  decoupled  from  the  real  time  battle  and  is  not  well 
suited  to  a  dynamic  battlespace. 

How  can  we  take  advantage  of  improved  situational  awareness  and  understanding  to  achieve  effective 
capability  for  both  proactive  and  reactive  decision  making  and  to  move  away  from  the  rigid  process-driven 
decisions?  That  question  is  the  focus  of  this  part  of  the  two-part  lecture. 

As  a  first  step,  let  us  look  at  the  role  that  information  plays  in  the  decision  process.  We  take  it  on  faith  that 
more  information  is  good,  and  less  is  bad,  but  that  may  not  really  be  true.  If  additional  information  causes  a 
cluttered  or  confused  view  of  the  situation,  then  we  are  better  off  without  it.  We  need  the  “right”  information, 
not  just  all  available  information.  This  is  indicated  in  Figure  2.  The  main  point  to  bear  in  mind  is  that  the 
information  is  presented  to  the  user  for  only  one  purpose:  to  assist  in  understanding  the  situation  and 
responding  to  it.  Modem  command  and  control  information  systems  are  focused  on  that  objective  and  include 
a  blend  of  technology  and  human  factors  that  support  the  cognitive  process  leading  to  decisions. 
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The  Use  of  Information  in  Decision-making 

•  The  More  We  Know  About  the  Situation,  the  Better  Is  Our 
Ability  to  Make  a  Good  Decision 

-  Not  Really  True:  the  More  We  Understand,  Not  the  More  We  Know 

-  Too  Much  Information  Can  Be  Worse  Than  Too  Little 

-  “Glare  of  War”  Combined  With  “Fog  of  War”  Can  Be  a  Problem 

•  Decision-support  Involves  the  Use  of  Information  to  Support 
Deliberate  Action 

-  The  Most  Important  and  Often  Overlooked  Aspect  of  Information 
System  Design  and  Operation 

-  If  We  Cannot  Act  on  the  Information,  Why  Bother  With  It? 

-  Timeliness  and  Relevance  Are  Key  to  Success 


Figure  2.  The  Relationship  Between  Information  and  Decision-Making 

The  previous  lecture  on  situational  awareness  and  understanding  discussed  a  simple  model  for  decision 
making.  We  saw  that  perception  is  the  basis  for  understanding  and  that  perception  was  a  complex  mixture  of 
knowledge,  confidence,  experience,  and  context. 

A  major  deficiency  in  that  model  is  that  it  implies  that  understanding  leads  directly  to  decision  making.  That 
is  not  correct.  The  decision  maker  must  first  interpret  his  or  her  understanding  as  a  set  of  alternative  courses 
of  action.  Then  he  or  she  must  be  able  to  select  the  best  course  of  action  from  those  alternatives. 

Clearly,  this  progression  from  understanding  to  decision  involves  a  relatively  complex  interaction  of  the  user’s 
perception,  his  or  her  experience  and  judgment,  and  a  further  understanding  and  interpretation  of  objectives. 
As  we  move  away  from  the  rigid  process-driven  decision  making  and  toward  more  dynamic  decision  making 
based  on  real  time  situational  understanding,  those  parts  of  the  process  that  rely  on  interpretation  and  j  udgment 
become  very  important.  The  decision  maker  will  need  to  be  able  to  take  initiative  to  interpret  guidance, 
constraints,  and  operational  objectives  in  terms  of  the  immediate  situation  in  the  battlespace.  The  tactical 
decision  aids  will  need  to  provide  support  attuned  to  those  j  udgmental  and  inferential  processes. 

Current  command  and  control  theory  lacks  a  clear  definition  of  the  part  of  the  cognitive  process  between 
understanding  and  decision.  It  is  sometimes  called  “wisdom”,  but  that  term  does  not  really  describe  it. 
Perhaps  a  better  term  would  be  “appreciation”  or  “military  appreciation”.  That  term  is  a  traditional  one  that 
means  a  comprehension  of  the  implications  of  the  situation  upon  an  ability  to  achieve  operational  objectives. 
This  is  precisely  the  type  of  comprehension  that  builds  upon  understanding  to  reach  the  decision,  so  we  have 
good  reason  to  adopt  it  and  insert  it  into  our  model. 
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The  model  for  decision  making  can  now  be  redrawn  to  include  “appreciation”  as  shown  in  Figure  3. 
Appreciation  is  “fed”  by  understanding;  supported  by  experience  (including  judgment  and  wisdom  within  the 
brain  of  the  decision  maker);  and  guided,  focused,  or  limited  by  operational  objectives,  priorities,  doctrine,  and 
so  forth.  The  output  of  appreciation  is  the  judgmental  determination  of  the  course  of  action  to  be  taken.  This 
is  the  decision. 


A  New  View  of  the  Decision  Process 


Objectives,  Priorities,  Doctrine,  TTPs,  Constraints 


Figure  3.  Basing  a  Decision  Upon  an  Appreciation  of  the  Situation 

Tactical  decision  aids  will  need  to  help  the  decision  makers  correlate  all  of  these  factors  quickly, 
unambiguously,  and  with  sufficient  confidence  to  make  the  decision.  One  of  the  major  complications  at  this 
stage  is  that  the  decision  maker  is  being  asked  to  make  a  judgmental  decision  with  potentially  lethal 
consequences  and  that  the  decision  needs  to  be  based  on  a  perception  of  the  situation  that  is  statistical  in 
nature.  We  do  not  have  the  advantage  of  being  able  to  feed  deterministic  inputs  into  a  well  defined  algorithm 
to  produce  a  deterministic  result.  Instead,  we  have  to  use  a  perception  of  the  situation  that  includes 
information  of  varying  degrees  of  clarity  and  credibility,  and  we  have  to  use  a  process  that  is  largely 
judgmental  to  decide  on  a  course  of  action  that  appears  to  satisfy  the  objectives,  rules  of  behavior,  and 
constraints. 

A  process  model  for  the  development  of  a  military  appreciation  is  illustrated  in  Figure  4.  This  is  a  notional 
model  for  the  development  of  the  military  appreciation.  The  input  information  is  on  the  left  hand  side,  feeding 
into  the  process  node.  The  supporting  resources,  judgment  and  vision,  are  shown  feeding  in  from  the  bottom. 
The  constraints  feed  in  from  the  top;  and  the  output  is  to  the  right. 
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Appreciation 

•  Appreciation  Is  the  Recognition  of  What  the  Situation  Means  in  Terms  of  the 
Response  Needed  to  Achieve  Objectives. 

•  The  Military  Appreciation  Is  the  Translation  of  Situation  Understanding  to 
Decision  Criteria  That  Include  Full  Consideration  of  Commander’s  Intent, 
Doctrine,  Rules  of  Engagement,  Physical  and  Environmental  Factors,  Tactics, 
Techniques,  and  Procedures,  Sustainment  Capabilities,  Enemy  Capabilities 
and  Intents,  and  All  Other  Factors  That  Influence  the  Course  of  Action. 

•  The  Military  Appreciation  Relies  on  Judgment  and  “Wisdom”  Based  on  the 
Full  Set  of  Education  and  Experience  of  the  Decision  Makers. 


Understanding  of 
The  Situation 


Objectives  and 
Commander’s  Intent 


Doctrine,  ROE, 
TTPs,  Etc. 


Environmental 
and  Physical  Factors 


Intelligence  on  Enemy 
Capabilities  and  Intents 


Priorities 

Sustainment  Factors 
Time  Constraints 
Etc. 


Developing  the 
Military  Appreciation 

T 

Judgment 
Wisdom 


>  Decision 

For  Example: 

•Move 

•Strike 

•etc. 


Figure  4,  A  Process  Model  for  Developing  the  Military  Appreciation 

The  current  state  of  the  art  in  information  technology  can  do  quite  a  bit  to  help  decision  makers  appreciate  the 
situation,  but  true  “artificial  intelligence”  remains  in  the  realm  of  science  fiction.  Computers  are  very  good  at 
adding,  subtracting,  and  comparing  numbers,  and  they  can  execute  processes  with  mathematical  precision,  but 
they  do  not  think.  If  the  decision  making  is  of  a  form  conducive  to  algorithmic  computation,  such  as 
development  of  a  fire  control  solution,  computers  are  far  better  than  people.  If  the  decision  making  requires 
judgmental  inferences  and  tradeoff  in  a  complex  situation,  the  human  beings  probably  outperform  the 
computers.  Such  is  the  case  with  most  tactical  decision  support  systems.  The  human  being  is  the  decision 
maker,  and  the  computer  is  the  support  tool.  Sometimes  it  can  actually  recommend  a  course  of  action  and 
develop  an  initial  draft  of  the  plan,  and  at  other  times  it  simply  does  the  bookkeeping  and  arithmetic. 

Collaboration  among  human  beings  is  often  required  when  the  appreciation  requires  different  sets  of 
experiences  and  skills.  For  example,  an  air  strike  on  a  ground  target  may  require  judgment  on  weapons 
effects,  threats,  weather,  and  so  forth.  In  those  cases,  collaboration  among  experienced  people  is  needed.  This 
can  be  either  in  a  single  physical  location  or  distributed  across  a  network.  Modern  technology  gives  us  the 
ability  to  construct  such  distributed  virtual  staffs  and  to  have  opportunity  to  use  expertise  from  the  finest  minds 
available  rather  than  j  ust  the  collocated  staff. 


Modem  Information  Technology  Can  Support 
Distributed  Virtual  Staffs 


Figure  5.  A  Collaborative  Map  Display  With  Overlays  Managed  by  Multiple  Users 

Figure  5  illustrates  a  form  of  distributed  collaboration  that  was  used  during  the  “Extending  the  Littoral 
Battlespace”  Advanced  Concept  Technology  Demonstration  (ELB  ACTD)  in  1999.  The  application  was  a 
commercial  product  called  CU-SeeMe.  It  allowed  users  at  various  locations  to  draw  on  a  shared  overlay 
superimposed  on  a  map  of  the  battlespace.  The  application  was  hosted  in  standard  desktop  and  laptop  PCs, 
connected  with  one  another  through  the  ELB  Wide  Area  Relay  Network  (WARNET).  In  this  way,  the  users  at 
different  locations  could  work  together  in  real  time  to  develop  the  maneuver  plan.  This  is  called  “synchronous 
collaboration”  because  all  the  users  are  collaborating  with  one  another  at  the  same  time. 

Another  form  of  distributed  collaboration  is  called  “asynchronous  collaboration”.  In  this  case,  each  user 
contributes  to  the  product  independently,  either  by  placing  products  on  a  shared  electronic  table  or  bulletin 
board,  or  by  e-mail  or  some  other  means  of  information  transfer.  When  we  use  e-mail  to  coordinate  drafts  of  a 
paper  or  briefing,  this  is  asynchronous  collaboration. 

Distributed  collaboration  during  the  ELB  ACTD  demonstration  in  1999  included  tactical  warfighters  in  a 
variety  of  environments,  not  just  people  at  consoles  in  command  posts  or  on  ships.  Figure  6  shows  a  few  of 
the  environments  during  that  demonstration.  The  command  center  was  onboard  the  USS  Coronado,  at  sea  off 
the  coast  of  San  Diego,  California.  The  furthest  reach  for  the  airborne  relay  network  (the  WARNET)  extended 
to  hand  held  computers  and  mobile  equipment  in  HMMWVs  in  Yuma,  Arizona.  Synchronous  collaboration 
using  CU-SeeMe,  video  and  voice  over  Internet  Protocol,  and  a  commercial  “chat  room”  application  was 
supported  and  allowed  distributed  tactical  decision  making  based  on  a  shared  view  of  the  situation. 
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The  Distributed  Collaboration  Can  Extend 
Down  to  the  Tactical  Level 


Figure  6.  Locations  of  Warfighters  During  the  Distributed  Collaboration  in  the  ELB  ACTD  Demonstration 

This  type  of  capability  will  become  more  important  in  the  future,  as  we  encounter  situations  in  which 
commander’s  intent,  priorities,  and  rules  of  engagement  will  have  to  be  interpreted  and  adapted  to  the  situation 
in  near  real  time.  A  full  appreciation  of  the  situation  will  demand  an  ability  to  consult  with  others  in  the  chain 
of  command  or  in  adjacent  commands  so  that  all  aspects  of  the  alternative  courses  of  action  can  be  considered. 
Tactical  decision  aids  will  need  to  include  capability  for  this  type  of  distributed  collaboration  and  consultation. 
They  will  have  to  be  truly  “network  centric”. 

We  also  need  to  remember  that  knowledge  and  understanding  of  the  situation  required  an  ability  to  present 
information  in  a  way  that  aligned  it  with  the  current  operational  context  and  integrated  it  to  form  recognizable 
patterns  and  relationships.  The  same  is  true  for  appreciation.  We  need  to  present  the  information  in  a  manner 
that  allows  the  user  to  view  objectives,  alternative  courses  of  action,  and  potential  enemy  actions  within  the 
framework  of  the  current  situation. 

The  tactical  decision  support  system  needs  to  help  users  who  are  likely  to  be  under  considerable  stress  at  the 
time.  A  human-system  interface  or  a  display  that  demands  too  much  attention  or  that  is  intolerant  of  input- 
output  errors  will  not  be  useful.  A  presentation  that  requires  the  user  to  integrate  and  correlate  in  his  or  her 
mind  will  be  less  useful  than  one  which  presents  a  clear  and  comprehensive  picture  of  the  important  aspects  of 
the  situation  and  the  implications  of  alternative  actions. 

The  presentation  should  also  allow  the  users  to  concentrate  on  the  job  at  hand  without  having  to  keep  track  of 
uncertainties  that  are  being  managed.  Automated  assistance  (software  agents)  can  reside  in  the  background  to 
track  those  types  of  things  and  to  cue  or  alert  the  user  when  necessary. 
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The  ELB  ACTD  demonstration  in  1999  showed  the  potential  value  of  a  tactical  data  network  and  decision 
support  system  operating  across  various  levels  of  command.  At  each  level,  the  users  have  a  specific  role  to 
play  in  the  decision  and  execution  process,  and  they  can  structure  their  displays  and  human-system  interfaces 
to  correspond  with  their  own  needs.  Software  agents  can  provide  cues  and  alerts  for  information  that  is  being 
tracked,  and  drilldown  is  available  when  details  are  needed.  However,  for  most  of  the  time  the  users  can  focus 
on  specific  information  associated  with  their  jobs. 


The  ELB  ACTD  Showed  The  Promise  of 
Modern  Technology  for  Tactical  Operations 
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Figure  7.  Tailored  Views  of  the  Situation  to  Adapt  to  the  Needs  of  Each  User 

Figure  7  illustrates  how  each  participant  interacts  with  shared  information  to  construct  an  appropriate  picture 
of  the  battlespace  and  to  reach  a  decision  motivated  by  an  observation  of  three  enemy  tanks.  The  decision  at 
the  lowest  level  is  to  report  the  observation.  At  one  or  two  levels  above  this,  the  decision  is  made  to  call  for 
fire.  A  decision  for  a  specific  indirect  fire  is  made  at  a  higher  level  in  this  example,  and  the  overall  direction 
of  maneuvers  and  fires  could  be  accomplished  through  a  collaboration  among  several  of  the  senior  levels  in 
the  tactical  chain  of  command. 
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Integrating  the  View  of  Situation,  Objectives 
and  Constraints  Helps  to  Support  the  Decision 


Logica’ s  DOHP  Overlay  on  a  US  Marine  Corps  C2PC  Situation  Display 
to  Show  NoGo  Lines  and  Excluded  Areas 


Figure  8.  Integration  of  Other  Decision  Factors  With  Observed  Situation  Information 

An  appreciation  of  the  situation  and  of  the  alternative  courses  of  action  is  supported  by  displays  that  show  how 
the  constraints  map  onto  the  battlespace.  Those  constraints  can  be  traffieability  affected  by  the  weather  or 
terrain,  or  they  could  be  due  to  rules  of  engagement  or  some  other  factors. 

Figure  8  is  a  screen-capture  from  the  Logica  UK  system  discussed  in  the  first  lecture.  It  shows  how  an  overlay 
of  maneuver  constraints  can  be  superimposed  on  the  map  used  to  plan  movements.  The  overlay  itself  can  be  a 
predefined  template  or  it  can  include  some  elements  that  are  calculated,  such  as  weather-related  traffieability 
or  inter-visibility  demands  for  covert  movement.  The  Logica  system  provided  software  agents  to  construct 
and  manage  these  overlays  and  provided  the  users  with  appreciation  of  where  they  could  move. 

The  tactical  decision  aid  can  also  advise  users  of  potential  courses  of  action  for  both  themselves  and  enemy 
forces.  The  overlay  shown  in  Figure  9  was  provided  in  the  Logica  system  to  show  where  units  of  various  sizes 
could  maneuver.  The  software  agent  that  Logica  used  was  based  on  a  standard  algorithm  for  maneuvering 
robots  in  spaces  with  obstructions.  Logica  adapted  this  algorithm  to  provide  reasoning  for  military 
organizations  moving  through  the  hilly  terrain  at  Camp  Pendleton,  California.  We  see  the  go,  no-go  areas  of 
the  prior  overlay,  and  we  see  the  allowable  maneuver  corridors  added  to  the  overlay. 
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Implications  of  the  Observed  Information  Can 
Be  Inferred  and  Integrated  in  the  Display 


Logica’s  Software  Agents  Construct  an  Overlay  That  Shows 
Maneuver  Corridors  for  Forces  of  Selected  Sizes  and  Compositions 


Figure  9.  Addition  of  Inferred  Knowledge  Can  Provide  Further  Enhancement  to  Military  Appreciation 

The  addition  of  the  maneuver  corridors  (lines  added  to  the  map  in  Figure  9)  gives  the  user  an  appreciation  for 
actions  that  the  enemy  may  take  as  well  as  his  or  her  own  allowed  actions. 

The  power  of  this  technology  is  multiplied  by  orders  of  magnitude  when  we  share  the  overlay  via  the  tactical 
data  network.  Then  we  can  consult  and  collaborate  to  come  to  a  decision  on  how  to  move  ourselves  in  concert 
with  the  other  elements  of  the  force.  We  also  have  a  means  to  do  real  time  adaptation  of  maneuvers  and  fires 
and  to  update  plans  to  match  the  situation. 

Planning  and  selection  of  a  course  of  action  is  only  one  part  of  the  control  and  direction  of  tactical  forces.  The 
tactical  decision  support  systems  must  also  provide  an  ability  to  monitor  and  manage  execution  as  the  battle 
unfolds. 

The  Office  of  Naval  Research  and  the  Naval  Air  Systems  Command  sponsor  a  program  at  the  Space  and  Naval 
Warfare  Systems  Command  (SPAWAR)  Systems  Center  in  San  Diego,  California  (SSC  SD)  to  develop  and 
deploy  new  capabilities  for  real  time  targeting,  retargeting,  and  execution  management  for  tactical  aircraft. 
The  program  is  called  “Real  Time  Execution  Decision  Support”  (REDS).  REDS  provides  a  variety  of 
software  tools,  displays,  and  interactive  query  capabilities  to  allow  the  users  to  designate  targets,  develop 
strike  packages  and  mission  plans,  and  direct  execution,  including  real  time  mission  monitoring  and  plan 
adjustment.  Operational  prototypes  are  being  deployed  with  Navy  carrier  air  wings  to  test  and  evaluate  the 
technology.  An  example  of  a  REDS  display  is  shown  in  Figure  10. 
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REDS  Tactical  Decision  Aid 


Figure  10.  Decision  Support  for  Real  Time  Planning,  Replanning,  and  Monitoring  of  Execution 

The  REDS  tactical  decision  aid  provides  a  view  of  the  battlespace  that  shows  both  friendly  and  threat 
capabilities  related  to  the  tactical  air  missions.  These  include  radar  coverage,  missile  envelopes,  targets,  and 
other  features  that  the  users  need  to  plan  missions  and  reach  conclusions  on  best  alternatives  for  engaging  time 
critical  targets  in  high  threat  environments.  With  such  a  tool,  the  mission  planners  can  designate  ingress  and 
egress  routes  to  avoid  threats  and  can  determine  requirements  for  defense  suppression,  electronic  warfare,  and 
other  support  to  the  mission. 

A  need  for  tactical  decision  support  does  not  end  when  the  mission  is  planned,  assigned,  and  launched.  Ability 
to  monitor  progress  and  to  adapt  to  unplanned  contingencies  is  also  needed. 

The  technique  used  in  REDS  is  shown  in  Figure  11.  It  provides  a  display  of  the  plan  in  terms  of  the  desired 
sequence  of  actions  and  the  specific  times  at  which  they  occur.  Those  are  the  horizontal  bars  in  the  display. 
Each  bar  represents  activities  that  are  supposed  to  occur  at  specific  times.  The  vertical  line  in  the  display  is  an 
indication  of  current  time.  It  moves  across  the  chart  from  left  to  right  as  time  progresses.  As  it  passes  through 
the  horizontal  bars,  they  are  assigned  colors,  from  red  to  green,  depending  on  whether  the  planned  status  at 
that  time  has  actually  been  achieved. 

This  type  of  display  is  a  very  user-friendly,  intuitive  way  for  the  tactical  decision  makers  to  view  the  entire  set 
of  missions  and  to  identify  where  they  are  on-track  or  in  trouble.  This  gives  a  very  clear  and  quick 
comprehension,  or  appreciation,  of  the  evolving  situation  with  respect  to  the  specific  user’s  functional  areas  of 
responsibility. 
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REDS  Real-Time  Mission  Monitor 
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Figure  1 1 .  REDS  Mission  Monitoring  Display 

Other  important  factors  in  making  a  tactical  decision  are  the  specific  empowerments  and  limitations  of 
alternative  actions  specified  by  rules  of  engagement  (ROE).  ROE  are  developed  at  the  highest  levels  of 
command  and  forwarded  downward  as  text  messages.  At  each  level,  the  ROE  can  be  expanded  with  further 
detailed  guidance  and  interpretation,  but  most  of  this  is  still  by  text  messages.  As  a  result,  the  warfighters  are 
burdened  with  manual  processing  of  textual  information  and  trying  to  keep  track  of  the  rules  as  they  try  to 
reach  a  decision. 

ONR  sponsors  an  effort  at  the  SPAWAR  Systems  Center,  San  Diego  (SSC  SD)  to  automate  the  ROE 
development,  dissemination,  and  application  process.  The  ROE  would  still  be  developed  and  approved  by 
commanders  and  their  staffs,  but  the  handling  of  this  information  would  now  be  done  much  more  efficiently 
and  in  a  manner  more  suited  to  time  critical  decision  making  under  stress.  The  ROE  information  would  be 
entered  into  data  bases  in  the  tactical  command  and  control  systems,  and  the  tactical  situation  data  bases  would 
have  automated  connections  to  the  appropriate  ROE  information. 

Figure  12  shows  a  hypothetical  display  of  a  tactical  situation  with  the  automated  ROE  augmentation.  The 
formatted  text  box  at  the  upper  right  hand  comer  of  the  screen  tells  the  operator  that  ROE  for  attack  have  been 
satisfied.  This  is  indicated  very  clearly  as  “ATTACK:  YES”. 
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Figure  12.  Rules  of  Engagement  Integrated  Into  the  Tactical  Situation  Display 

If  the  operator  needs  to  see  the  specific  rationale  for  this  decision  that  ROE  have  been  satisfied,  he  or  she  can 
drilldown  and  request  those  details.  Drilldown  information  is  included  in  the  display  shown  in  Figure  13. 

The  ROE  explanation  box  now  shows  the  information  that  supported  the  conclusion  that  an  attack  was 
authorized  under  the  current  rules  of  engagement,  based  on  specific  observations.  The  conclusion  that  the 
ROE  allow  attack  (ATTACK:YES)  is  based  on  two  specific  observables  that  satisfy  that  ROE.  Those  are 
noted  as  “Cl:  In  No  Fly  Zone”  and  “C2:  Target  has  previously  attacked”. 

The  operator  can  conclude  very  quickly  and  unambiguously  that  the  two  criteria  specified  in  the  ROE  have 
been  satisfied  and  that  the  attack  is  permitted.  This  also  provides  a  permanent  record  of  the  basis  for  the 
decision,  which  can  be  very  useful  for  after-action  evaluations,  and  also  for  legal  considerations  if  such  are 
necessary. 

Remember  that  the  appreciation  of  the  situation  usually  leads  to  a  judgmental  decision  with  potentially  lethal 
consequences.  The  ability  to  make  such  judgments  is  often  improved  if  the  decision  maker  can  be  sure  that  a 
clear  and  unambiguous  basis  exists  for  the  judgment  and  that  clear  documentation  of  that  basis  will  be 
preserved. 
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Figure  13.  ROE  Display  With  Additional  Drilldown  Details 

Advanced  information  technologies  such  as  those  discussed  here  will  cause  major  changes  in  command  and 
control  systems  and  processes.  One  of  these  changes  is  a  shift  from  fully  manual  processes  to  a  reliance  on 
automation  to  support  the  human  beings,  and  even  toward  fully  automated  processes. 

Automation  is  most  important  for  tactical  combat  systems  and  weapon  control  systems,  where  complexity  and 
time  demands  make  it  difficult  for  human  beings  to  be  directly  in  the  control  loop.  Even  the  act  of  flying  an 
aircraft  is  sometimes  beyond  the  capability  of  human  beings.  Modem  tactical  fighter  and  attack  aircraft  are 
often  highly  unstable  platforms.  This  allows  them  to  have  very  fast  response,  but  it  also  makes  them  very 
difficult  to  fly  by  direct  interaction  of  the  pilot  with  flight  control  mechanisms.  Consequently,  a  computer  flies 
the  aircraft,  and  the  pilot  exercises  control  of  the  computer,  to  provide  specific  objectives  for  it  to  meet.  This 
is  an  example  of  a  human-supervised  automated  process. 

Tactical  combat  direction  and  command  and  control  offer  more  opportunity  for  human  beings  to  act  within  the 
decision  loop.  Here  the  demands  of  time  compression  and  complexity  are  still  large,  but  the  human  beings  are 
able  to  cope  with  them  as  long  as  the  automated  systems  provide  them  with  timely  and  “user  friendly”  support. 
Systems  such  as  the  Logica  tactical  decision  aid,  REDS,  and  automated  ROE  are  examples  of  important  efforts 
to  provide  such  support  to  users  who  are  embedded  in  the  tactical  decision  processes. 

Figure  14  shows  the  wide  variety  of  human-computer  relationships  that  occur  in  command  and  control  systems 
and  processes. 
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The  Relationship  Between  Humans  and 
Machines  Is  Changing 

•  Manual  Process 

-  People  Do  the  Work  Using  Pencil,  Paper,  Telephone,  etc. 

•  Computer-Aided  Manual  Process 

-  Humans  Use  the  Technology  to  Do  the  Bookkeeping  and  the 
Mathematics  and  to  Move  the  Information  Among  Themselves 

•  Automated  Process 

-  Computers  Run  the  Show 

•  Human-Supervised  Automated  Process 

-  Computers  Run  the  Show,  but  Humans  Can  Monitor  and  Intervene 
Directly  in  the  Process 

•  Human-in-the-Loop  Process 

-  The  Human  Is  Inserted  As  a  “Functional  Module”  Within  the 
Automated  Process 

•  “Mixed  Initiative”  Process 

-  The  Human  and  the  Computer  Do  Interconnected  Tasks 


Figure  14.  The  Wide  Spectrum  of  Human-Computer  Relationships  in  Command  and  Control 

Architecture,  commonality,  and  interoperability  are  additional  factors  that  need  to  be  taken  into  account  when 
building  a  command  and  control  system.  At  the  tactical  level  of  command  and  control,  we  are  often 
challenged  by  competing  demands.  On  the  one  hand,  we  want  to  make  use  of  common  systems  to  the  greatest 
extent  possible  to  reduce  burdens  of  training,  operation,  and  maintenance.  It  also  allows  us  to  reduce  total  cost 
of  ownership  by  using  commercial-off-the-shelf  (COTS)  equipment  rather  than  expensive,  purpose-built 
military  hardware  and  software.  When  requirements  are  not  oriented  on  near  real  time  control,  general 
purpose  systems  are  often  appropriate. 

Unfortunately,  tactical  systems  can  impose  very  stressing  requirements  for  near  real  time  response  and 
software  trustworthiness  that  often  demand  purpose-built  technology.  We  do  not  rely  on  a  COTS  personal 
computer  and  general  purpose  software  to  generate  fire  control  solutions  within  a  second  or  less;  and  we  do 
not  want  that  COTS  computer  and  software  guiding  the  weapon  in  flight.  For  these  functions,  we  generally 
embed  purpose-built  computing  in  the  combat  systems  and  weapon  systems. 

Tactical  decision  aids  are  at  the  unfortunate  “seam”  between  technology  based  on  general  purpose  and 
purpose-built  computing.  We  will  often  find  hybrid  systems,  where  part  of  the  process  is  based  on  COTS  and 
part  is  purpose-built  for  the  military  task.  This  could  change  as  the  performance  and  trustworthiness  of  COTS 
improves,  but  we  will  have  to  live  with  the  current  situation  until  that  time. 


Figure  15  summarizes  considerations  of  commonality  versus  purpose-built  “stovepipe”  systems. 
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An  Important  Architectural  Issue: 
“Stovepipes”  Versus  Commonality 

•  A  Trade-Off  Between  Objectives  That  Are  Often 
Competing 

-  Tailored,  Optimized  Performance  (e.g.  For  Weapons  Control) 

•  Real-Time 

•  Strongly  Focused  on  a  Specific  Function 

-  General  Purpose,  Common  Systems  and  Applications 

•  Economy  of  Scale  for  Procurement  and  O&M  Support 

•  Assured  Interoperability 

•  One  Size  Does  Not  Necessarily  Fit  All 

-  Administrative  Functions 

-  Strategic  Command 

-  Operational  and  Tactical  Command 

-  Tactical  Control/Combat  Direction 

-  Combat  Systems  and  Weapons  Systems 


Figure  15,  Considerations  in  Building  for  Commonality  or  Purpose-Built  Functionality 

No  discussion  of  situational  awareness  and  tactical  decision  aids  would  be  complete  without  some  reference  to 
interoperability.  This  is  especially  true  for  coalition  operations,  where  differences  arise  not  only  with  the 
technologies  but  also  with  operational  concepts,  procedures,  and  language. 

The  first  three  “bullets”  in  Figure  16  are  the  interoperability  areas  that  we  are  used  to  seeing.  Once  we  achieve 
this  level  of  interoperability,  we  can  exchange  data  and  process  information.  Our  systems  will  work  together 
at  least  to  that  extent. 

The  next  two  “bullets”  address  the  parts  of  the  process  that  enable  knowledge,  understanding,  and 
appreciation.  If  we  do  not  recognize  semantic  differences,  we  may  misinterpret  information  and  therefore 
make  an  error  perception.  If  our  responses  to  information  are  not  common,  we  may  make  decisions  that 
conflict  with  other  decisions  in  the  coalition  force. 

These  last  two  “bullets”  are  the  real  challenge  in  coalition  warfare.  We  know  how  to  achieve  interoperability 
up  to  the  application  level,  and  we  even  have  some  capabilities  up  to  the  semantic  level,  but  we  lack  the 
fundamental  tools  to  achieve  the  higher  levels.  This  is  a  major  focus  area  for  current  science  and  technology 
programs. 
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A  Word  About  Interoperability 
What  Does  It  Really  Mean? 

•  Technical  Interoperability 

-  The  Hardware  and  Software  Can  Operate  Together 

•  Transaction  Interoperability 

-  Information  Can  Be  Exchanged  and  Processed 

-  Protocols  Are  Consistent  Across  the  Systems 

•  Application  Interoperability 

-  The  Applications  Can  Exchange  Information 

•  Semantic  Interoperability 

-  Language  and  Meaning  Are  Commonly  Understood 

•  Response  Interoperability 

-  The  Expected  Responses  to  Information  Are  Understood  by  All 

-  e.g.  We’ve  Heard  the  Joke  About:  “Secure  the  Building” 


Figure  16.  Interoperability 

In  summary,  the  movement  toward  “information-leveraged”  warfare,  whereby  the  warfighters  achieve  greater 
effectiveness  through  better  use  of  information,  has  both  positive  and  negative  aspects. 

We  saw  in  the  first  lecture  that  the  “fog  of  war”  may  be  penetrated  to  some  degree,  but  it  will  still  exist.  We 
also  saw  that  the  technology  that  helped  penetrate  the  “fog”  presented  us  with  potential  overload,  the  “glare  of 
war”. 

In  this  second  lecture  we  saw  that  a  major  challenge  in  modern  warfare  is  to  be  able  to  respond  very  quickly  to 
complex  situations  and  to  reach  a  military  appreciation  and  a  decision  based  on  our  appreciation  of  the 
situation.  This  caused  us  to  be  concerned  with  making  j  udgments  based  on  imprecise  information,  where  lack 
of  credibility  and  confidence  can  be  impediments  to  decision  making.  We  saw  that  some  form  of 
“partnership”  between  the  human  beings  and  the  technology  will  be  required  as  we  move  more  and  more 
toward  the  real  time  tactical  functions  or  toward  the  massive  data  base  environments  of  the  higher  level 
operational  and  strategic  commands. 


These  challenges  provide  a  rich  context  for  current  research  and  development  for  situational  awareness  and 
understanding  and  tactical  decision  aids. 
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Mission  Management  and  Crew  Assistance  for  Military  Aircraft 
-  Cognitive  Concepts  and  Prototype  Evaluation  - 

Dr.  Axel  Schulte 
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1  SUMMARY 

This  paper  describes  an  approach  to  cognitive  and  co-operative  operator  assistance  in  the  field  of  tactical  flight 
mission  management.  A  framework  for  a  generic  functional  concept  is  derived  from  general  considerations  of 
human  performance  and  cognitive  engineering.  A  system  built  according  to  these  human-centred  design 
principles  will  be  able  to  keep  up  with  the  change  of  situation  parameters,  in  order  to  provide  situational 
adapted  operator  assistance.  Such  a  Cognitive  Assistant  System  (e.g.  Onken  &  Walsdorf,  2000)  represents  an 
approach  to  ensure  the  highest  degree  possible  of  situation  awareness  of  the  flight  deck  crew  as  well  as  a 
satisfactory  workload  level. 

This  generic  approach  to  mission  management  and  crew  assistance  for  military  aircraft  has  been  realised  in 
different  application  domains  such  as  military  transport  and  air-to-ground  attack.  Even  applications  in  the 
domain  of  unmanned  aerial  vehicles  (UAV)  (Stutz  et  al.,  2001)  are  in  reach.  This  paper  mainly  covers  two 
state-of-the-art  research  and  development  activities:  the  Crew  Assistant  Military  Aircraft  (e.g.  Schulte  &  Stutz, 
1998;  Frey  et  al.,  1999;  Stutz  &  Schulte,  2000)  is  a  functional  prototype  for  the  air  transport  application  and 
the  Tactical  Mission  Management  System  (e.g.  Schulte  et  al.,  1999;  Schulte  et  al.,  2001/a&b)  as  an 
experimental  solution  for  air-to-ground  attack  aircraft.  The  paper  gives  details  on  the  prototype  development 
and  the  experimental  evaluation. 

2  INTRODUCTION 

Performing  military  combat  missions  in  an  uncertain  dynamic  tactical  environment,  presents  a  potentially 
intolerable  workload  for  the  crew.  Therefore,  several  research  and  development  activities  are  conducted  in 
order  to  automate  flight  crews’  tasks.  Figure  1  illustrates  the  expected  benefit  of  an  increase  of  complexity  of 
conventional  automation  as  opposed  to  a  so-called  cognitive  automation  approach  (e.g.  Onken  Sc  Walsdorf, 
2000;  Putzer  &  Onken,  2001).  Investigations  of  modem  aircraft  cockpits  show  that  a  further  increase  in  use  of 
conventional  automation  will  not  necessary  result  in  increased  productivity.  Automation  itself  became  a 
complex  element  within  the  already  complex  environment  of  the  cockpit.  In  some  cases  conventional 
automation  has  even  become  the  key-factor  for  decreased 
safety  (e.g.  due  to  ‘mode  confusion’),  (e.g.  Wiener  Sc 
Nagel,  1988)  The  reason  for  this  seems  to  be  found  in  the 
unpredictability  of  the  machine’s  behaviour  due  to 
inconsistencies  between  the  machine  function  and  the 
pilot’s  mental  model  of  it. 

Consequently,  Billings  (1991  &  1997)  stated  his  well 
known  principles  of  human-centred  automation  i.e.: 

•  The  pilot  bears  the  responsibility  for  safety  of  flight. 

•  Pilots  must  remain  in  command  of  their  flights. 

And  as  corollaries: 

•  The  pilot  must  be  actively  involved. 

•  The  pilot  must  be  adequately  informed.  Figure  1:  Expected  benefit  of  conventional 

•  The  operator  must  be  able  to  monitor  the  automation  and  cognitive  automation 

assisting  them. 
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•  The  automated  systems  must  be  therefore  predictable. 

•  The  automated  systems  must  also  monitor  the  human  operator. 

•  Every  intelligent  system  element  must  know  the  intent  of  other  intelligent  system  elements. 

These  principles  imply  the  co-operative  approach,  establishing  the  automation  as  a  team  player  within  the 
flight-deck  crew.  But,  how  can  this  practically  be  accomplished?  Onken  (1994)  formulates  his  basic 
requirements  concerning  human-machine  interactions  in  combat  aircraft  as  follows  (cited  from  Onken  & 
Walsdorf,  2000): 


(1)  It  must  be  ensured  the  representation  of  the  full  picture  of  the  flight  situation,  including  that  the  attention 
of  the  cockpit  crew  is  guided  towards  the  objectively  most  urgent  task  or  sub-task  as  demanded  in  that 
situation. 

(2)  A  situation  with  overcharge  of  the  cockpit  crew  might  come  up  even  when  situation  awareness  has  been 
achieved  by  the  pilot  crew.  In  this  case  the  assistant  system  has  to  transfer  the  situation  into  a  normal  one 
which  can  be  handled  by  the  crew  in  a  normal  manner. 

The  basic  requirement  (1)  addresses  the  system  performance  needed  to  most  effectively  ensure  the  pilots’ 
situation  awareness.  The  basic  requirement  (2)  is  formulated  to  avoid  pilots’  overcharge,  in  particular  as  to 
planning,  decision-making,  and  plan  execution  tasks. 


The  actual  automation  functions  required  to  provide  intelligible  interaction  and  efficient  use  of  such  a 
cognitive  assistant  system  are  derived  from  a  concept  taking  into  consideration  the  process  of  human 
information  processing,  the  cognitive  process  (as  described  in  the  following  chapter).  The  approach  aims  at 
the  provision  of  crew  assistant  functions  which  focus  upon  the  interpretation  and  diagnosis  of  the  situation  and 
the  monitoring  and  if  necessary  the  retrieval  of  the  integrity  of  superior  goals  such  as  safety,  combat  survival 
and  mission  accomplishment  (Schulte  &  Stutz,  2000).  In  order  to  realise  this  approach,  a  way  had  to  be  found 
how  to  deal  with  human  goal  knowledge  in  machine  systems.  Today’s  cockpit  avionics  completely  neglect 
this  kind  of  information.  Of  course,  the  treatment  of  goals  is  implicit  (e.g.  GCAS  is  designed  to  avoid  ground 
collision  per  definition).  But,  goals,  goal  violations  and  the  possible  interference  of  different  goal  domains 
(e.g.  combat  survival  vs.  mission  accomplishment)  are  not  yet  processed  explicitly.  In  most  cases  conventional 
automation  is  used  in  more  or  less  separated  systems.  Coupling  of  application  domains  (e.g.  ground  collision 
and  threat  exposure)  has  to  be  performed  by  the  human  operator  on  his  own.  Cognitive  systems  promise  the 
exploitation  of  synergetic  resources  within  the  co-operation  of  machine  system  and  human  operator.  (See 
Figure  2.) 
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Figure  2:  Synergetic  resources  to  be  exploited  through  man-machine  co-operation 

The  next  chapter  deals  with  the  refinement  of  such  a  cognitive  automation  concept  on  the  basis  of  the 
consideration  of  the  human  knowledge  processing  scheme.  A  generic  functional  framework  will  be  derived. 
The  following  chapter  provides  the  results  of  two  research  and  development  programmes  being  based  upon 
these  concepts. 
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3  COGNITIVE  CONCEPTS 

In  the  introduction  the  issue  of  changing  from  a  conventional  automation  philosophy  towards  a  more 
advanced  and  promising  concept  in  terms  of  safety  and  productivity  has  been  just  scratched  on  the  surface. 
The  following  sub-sections  are  meant  to  discuss  the  cognitive  design  philosophy  in  a  more  systematically 
manner. 

The  first  issue  to  be  considered  is  the  paradigm  of  migration  from  conventional  automation  towards  a  co¬ 
operative  architecture.  The  work  process  levels  in  order  to  classify  the  cockpit  tasks  are  defined  having 
Rasmussen’s  (1983)  human  performance  levels  in  mind.  Co-operative  work-share  between  human  and 
machine,  yet,  demands  advanced  capabilities  on  the  machine  side. 

In  a  next  step  the  human  problem-solving  strategy  and  the  cognitive  process  are  considered  in  order  to  form  a 
framework  for  the  definition  of  such  machine  functions  dedicated  to  the  establishing  of  a  close -partner  work 
relationship  between  the  automation  and  the  human  operator.  Again,  a  model  of  the  human  problem  solving 
strategy  based  upon  Rasmussen’s  work  (1983  &  1986)  is  used  to  identify  current  shortfalls  in  automation. 

Finally,  the  approach  of  goal  handling  within  the  machine  system  is  treated  in  order  to  enable  the  machine  to 
interact  on  a  knowledge  based  cognitive  level.  A  model  of  superior  goals  and  the  deduction  of  resulting  tasks 
is  given. 

3.1  Co-operative  automation 

The  variety  of  tasks  to  be  performed  by  the  pilot  during  a  tactical  flight  mission  result  in  a  workload  on  all 
work  process  levels  (see  Figure  3),  ranging  from  skill-based  manipulatory  control  (bottom/yellow)  through 
rule-based  system  interaction  (middle/green)  up  to  general  knowledge -based  problem  solving  tasks  (top/blue). 
Conventional  automation  traditionally  focuses  on  relieving  the  crews  from  exhausting  routine  actions,  thereby 
being  granted  full  autonomy  in  certain  well  defined  areas  (Figure  3,  left).  Expanding  this  strategy  of 
automation  into  task  domains  primarily  subjected  to  rule-  and  knowledge-based  crew  action,  leads  to  severe 
problems  in  the  area  of  man-machine  interaction  (Figure  3,  middle).  Significant  for  this  kind  of  development 
are  very  complex  avionics  structures  and  functions  taking  over  full  autonomy  for  comprehensive  parts  of  the 
flight,  while  reducing  the  pilot  to  a  mere  solver  of  abnormal  situations  (Hollnagel,  1995;  Billings,  1997). 
Therefore,  new  progressive  methods  are  demanded  when  it  comes  to  expand  automation  into  all  aspects  of 
flight  and  mission  management.  The  most  promising  way  to  proceed  is  the  concept  of  an  automated  system 
acting  in  a  co-operative  relation  rather  than  separating  the  crew  from  the  basic  aircraft  systems  (Figure  3, 
right).  Being  well  aware  of  the 
complex  task  to  perform,  the 
crew  interprets  the  output  of  the 
automation  system  as  the 
recommendation  of  an  additional 
electronic  crewmember.  The 
decision-making  to  accept  or 
reject  the  machine’s  advice  is 
allocated  to  the  crew.  Proceeding 
this  way,  the  crew  is  kept 
continuously  in  the  decision  loop, 
as  according  to  Billings’ 
principles,  and  is  able  to  employ 
the  full  strength  of  human 
performance.  At  the  same  time 
the  crew  takes  advantage  of  the 
particular  strengths  and  abilities 
of  the  system. 

Establishing  the  postulated  co-operative  task  allocation  within  a  close-partner  work  relationship  between  the 
human  operator  and  the  machine  requires  to  qualify  the  automated  system  to  be  an  equal  and  competent  team 
player,  as  one  would  expect  from  a  human  counterpart.  The  next  section  yields  a  closer  look  at  human 
performance  models  in  task  situations  to  be  used  as  a  paradigm  for  a  technical  system’s  human-like  behaviour. 
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3.2  Human  problem-solving  strategy  and  the  cognitive  process 

Considering  the  automated  functions  within  today’s  cockpit  avionics  and  flight  guidance  systems  against  the 
background  of  Rasmussen’s  model  of  the  human  knowledge  processing  scheme  (see  Figure  4)  shortcomings 
in  conventional  automation  design  can  be  easily  identified  (Onken  &  Walsdorf,  2000).  It  covers  only  about  the 
half  of  what  has  been  considered  the  spectrum  of  human  cognitive  functions.  This  situation  calls  for  action  in 
the  fields  of  a  profound  situation  recognition  and  identification  as  well  as  the  goal-driven  decision-making. 
This  scope  of  duties  is  in  compliance  with  Onken’s  basic  requirements  as  cited  above.  Cognitive  automation 
will  cover  the  whole  process  of  building  an  internal  comprehensive  representation  of  the  relevant  parts  of  the 
external  world  (i.e.  the  picture  of  the  situation),  a  type  of  activity  which  has  been  mostly  left  up  to  the  human 
operator,  so  far.  The  so-gained  situation  picture  will  be  the  basis  for  the  crucial  goal-driven  decision-making 
process  (Putzer  &  Onken,  2001). 


Conventional  Automation 


Figure  4:  Scope  of  cognitive  and  conventional  automation  within  the  Rasmussen  scheme  (1983) 

Besides  an  effective  means  of  communication,  a  prerequisite  to  establish  this  partnership  relation  between 
man  and  machine  is  the  implementation  of  transparent  functional  behaviour  within  the  automation  system.  In 
order  to  make  the  machine  output  easy  to  comprehend  and  evaluate,  and  to  establish  a  close-partner  work 
relationship,  both  crew  and  machine  have  to  reason  from  the  same  principles.  Thus,  the  analogue  problem¬ 
solving  strategies  and  mechanisms  have  to  be  implemented  in  the  automated  system  in  a  similar  way  to  that 
which  can  be  found  in  the  human  counterpart.  This  concept  is  the  core  element  of  cognitive  automation. 

Figure  5  yields  a  modified  view  on  the  elementary  steps  of  human  problem  solving  behaviour,  which  tries  to 
reflect  the  aspects  of  condensation  of  information  on  higher  processing  levels  and  the  closed-loop  system 
properties  of  the  human  being  feeding  back  to  his  environment.  This  behaviour  model  can  be  easily 
transferred  into  a  design  of  machine  functions  (Schulte  &  Stiitz,  2000). 

On  the  lowest  level  a  state -oriented  acquisition  of  environmental  signals  is  conducted  and  direct  manipulatory 
output  is  generated.  Problems  demanding  a  certain  amount  of  data  abstraction  and  knowledge  transfer 
typically  cannot  be  solved  on  this  level.  Therefore,  further  data  interpretation  is  necessary,  taking  into  account 
superior  context  knowledge.  A  task-related  aggregation  and  fusion  of  information  can  be  derived  as  a  result. 
On  the  structure-oriented  level  the  data  so-gained  is  further  processed  using  additional  rule  and  knowledge 
bases  in  order  to  reach  a  more  profound  problem  diagnosis  and  a  spectrum  of  possible  solutions.  Again,  back 
on  the  context-level,  a  decision  on  how  to  proceed  is  found  by  the  use  of  planning  and  forward-simulation 
results.  Then,  the  derived  solution  is  passed  to  the  state -level  for  execution,  thus  ensuring  successful  problem 
solving  under  consideration  of  all  relevant  circumstances  and  all  available  information. 

Simple  automation  implementations  within  clearly-cut  task  domains  such  as  auto-pilot  or  flight  director 
systems  usually  can  be  seen  as  immediate  instantiations  of  the  processing  steps  on  the  state-level,  which  are 
directly  connected  through  functional  relations.  Autonomous  planning  functions  are  state-of-the-art  in  today’s 
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Flight  Management  Systems.  Assisting 
the  crew  also  on  the  higher  levels  of 
their  problem  solving  tasks,  some 
abstract  and  therefore  more  versatile 
task  knowledge  has  to  be  made 
available  to  the  machine. 

The  cyclic  character  of  the  knowledge- 
based  information  processing  proce¬ 
dure  within  the  human  being  as  well  as 
implemented  in  a  machine  system  has 
been  brought  to  the  point  by  the 
research  group  around  Onken  (e.g. 
Walsdorf  et  ah,  1999;  Onken  & 
Walsdorf,  2000  and  many  other 
publications).  Derived  from  the 
recognition-act  cycle  they  construct  a 
comprehensive  model  of  the  cognitive 
process  or  cognitive  loop  (see  Figure 
6).  The  cognitive  loop  comprises  six 
cognitive  sub-processes  which  will  be 
run  through  continuously.  The 
cognitive  sub-processes  are  referred  to 
as: 

(1 )  data  acquisition  from  the  external  world, 

(2)  situation  interpretation, 

(3)  goal  activation  (i.e.  situation  diagnosis), 

(4)  planning  and  decision  making, 

(5)  scheduling  of  the  tasks  to  be  performed  and 

(6)  control  and  execution  of  the  derived  actions. 


Figure  5:  Human  cognitive  problem  solving  behaviour 


Figure  6:  Overview  of  cognitive  process  as  a  model  for  required  machine  functions  (Putzer  &  Onken ,  2001) 
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Besides  the  sub-processes  a  main  building  block  of  the  cognitive  loop  is  the  knowledge -trunk,  it  comprises 

•  static  background  (a  priori)  knowledge  relevant  for  the  application  domain,  and 

•  dynamic  situation  (a  posteriori)  knowledge  generated  as  an  output  of  the  cognitive  sub-processes  (i.e. 
mental  model  /  cognitive  yield). 

The  combination  of  both  as  well  as  the  relations  amongst  the  diverse  knowledge  objects  constitute  the  basis 
for  the  knowledge-based  situation  interpretation,  diagnosis,  decision-making  and  scheduling  processes.  Onken 
and  Walsdorf  (2000)  suggest  a  formal  treatment  of  the  situation  knowledge.  Putzer  &  Onken  (2001)  describe 
the  cognitive  sub-processes  (=  Transformators)  and  their  strict  object-oriented  implementation  in  more  detail. 


3.3  Goal  and  task  model 

As  the  structure  of  functions  stated  above  makes  obvious,  a  machine -immanent  representation  and  processing 
of  goals  and  tasks  is  compelling,  thus  establishing  a  common  ground  of  understanding  between  the  avionics 
system  and  the  crew.  Reasoning  from  first  principles,  the  machine  system  is  capable  of  inferring  the  same 
solutions  a  human  pilot  is  most  likely  to  reach  in  this  situation.  This  goal  model  considers  the  principal  kinds 
of  pilot  motivation  such  as  flight  safety,  combat  survival  and  mission  accomplishment  (see  Figure  7)  (Schulte 
et  al.,  1999;  Schulte  &  Stiitz,  2000). 


Figure  7:  Model  of  goals  and  tasks  for  conflict  detection 

The  primary  aim  of  the  introduction  of  the  goal/task  model  is  the  detection  and  prioritisation  of  conflicts  in  the 
mission  progress.  A  conflict  is  defined  as  the  violation  of  a  goal.  Starting  from  the  abstract  goal  classes, 
application-specific  sub-goals  are  derived  (e.g.  meet  a  given  time-over-target,  TOT).  The  parameterisation  of 
the  sub-goals  is  performed  using  the  mission  order  and  the  current  situation.  This  might  include  the 
application  of  scheduled  values  or  tolerances  and  even  the  de-activation  of  single  sub-goals  according  to  the 
current  flight  phase  (e.g.  goal  ‘meet  TOT’  is  no  longer  relevant  in  the  flight  phase  Ergress).  For  conflict 
detection  the  specified  sub-goals  (i.e.  current  tasks)  are  compared  with  the  actual  situational  parameters.  In  the 
case  of  an  intolerable  deviation  a  conflict  is  detected  and  can  be  passed  on  for  further  processing. 

This  approach  heavily  relies  on  its  capability  to  analyse  all  situational  elements  and  their  influence  on  the 
given  goal  structure.  Several  sub-functions  using  expert  knowledge  are  necessary  to  perform  the  needed 
interpretation  of  flight  progress,  environmental,  tactical  and  aircraft  data.  Only  when  this  overall  situational 
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picture  is  available  to  the  machine  system  it  is  possible  to  determine  and  prioritise  goal  violations  reliable  by 
the  centralised  goal  conflict  analysis. 


3.4  Determination  of  current  pilot  tasks 

Defining  the  relevant  goal  structure  for  finding  conflicts  and  appropriate  conflict  solutions  is  greatly 
influenced  by  the  aircraft’s  current  state  in  flight  progress.  Therefore,  the  capability  of  autonomous  flight 
phase  recognition  is  vital  to  the  system.  This  includes  the  determination  of  the  pilot’s  tasks. 


Figure  8:  Phase  of  flight  and  pilot  tasks  determination 

Figure  8  shows  the  architecture  of  the  continuous  phase  of  flight  (PoF)  and  pilot  tasks  determination.  The 
cross  in  Figure  7  marks  about  the  place  where  this  sub-system  has  to  be  inserted  into  the  goal  -  task  model. 
The  starting-point  of  the  algorithm  is  an  instantaneous  top-level  phase  of  flight  determination  based  upon  a 
heuristical  classification  approach.  The  PoF  Level  1  triggers  the  activation  of  one  of  the  phase  transition 
networks,  each  of  which  represented  by  a  State  Transition  Network  (STN).  The  inset  in  Figure  8  gives  an  idea 
of  such  a  network.  The  output  of  the  active  network  is  the  refinement  of  the  flight  phase.  The  PoF  Level  2 
controls  the  rule-based  inference  mechanisms  in  concurrent  task  domains  for  the  pilot’s  expected  tasks 
determination.  Thereby,  the  required  sub-goal  specifications  can  be  derived  in  order  to  determine  the  current 
tasks  for  monitoring  by  the  system. 

4  PROTOTYPE  EVALUATION 

After  having  formulated  the  cognitive  system  approach  in  the  field  of  crew  assistance  and  tactical  mission 
management,  this  chapter  deals  with  the  description  of  two  different  experimental  systems:  The  Crew 
Assistant  Military  Aircraft  (CAMA)  and  the  Tactical  Mission  Management  System  (TMM).  Both  prototypes 
were  developed  according  to  the  described  design  principles  of  cognitive  automation. 

4.1  Crew  Assistant  for  Military  Transport  Aircraft 

The  Crew  Assistant  Military  Aircraft  (CAMA)  provides  an  example  of  how  to  assist  military  cockpit  crews 
during  transport  missions.  CAMA  is  a  knowledge -based  cognitive  assistant  system  under  development  in 
close  co-operation  between  the  partners  ESG,  the  University  of  the  German  Armed  Forces,  the  German 
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Aerospace  Research  Establishment  (DLR),  and  EADS  Military  Aircraft  since  1993.  CAMA  is  the  latest 
development  within  the  Cockpit  Assistant  SYstems  family  derived  from  CASSY  (e.g.  Prevot,  et  ah,  1995). 

4.1.1  Prototype  implementation 

This  sub-section  will  give  a  brief  insight  into  the  functional  modules  of  CAMA.  Figure  9  shows  the  functional 
structure  of  CAMA.  it  basically  follows  the  generic  structure  of  the  cognitive  sub-processes  as  stated  before. 

In  order  to  ensure  situation  awareness  for  the  machine  system,  all  information  produced  by  the  modules  is 
assembled  in  a  Central  Situation  Representation  (CSR)  and  this  way  provides  a  complete  dynamic  database  of 
the  current  situation.  This  can  be  seen  as  an  analogue  to  the  pilot’s  own  mental  representation  of  the  actual 
situation.  Static  databases  contain  information  such  as  navigational,  terrain  and  feature  data.  The  CSR  forms 
the  body  of  the  cognitive  process  as  shown  in  Figure  6. 

Dynamic  external  data  such  as  aircraft  sensory  data,  air  traffic  control  and  C2  instructions  as  well  as 
environmental  information  are  gathered  via  an  external  communication  interface.  This  part  represents  the 
cognitive  sub-process  (1)  of  data  acquisition  from  the  external  world. 

Various  modules  provide  crucial  information  on  health  status  of  aircraft  systems  (SI),  environmental  aspects 
(El),  and  the  flight  progress.  The  Tactical  Situation  Interpreter  (TSI)  calculates  the  local  threat  distribution 
along  the  mission  plan.  In  this  way  CAMA  is  able  to  perform  conflict  detection  with  respect  to  local  changes 
in  the  tactical  situation.  The  modules  Pilot  Behaviour  Interpreter  (PBI)  as  well  as  Pilot  Intent  and  Error 
Recognition  (PIER)  serve  mainly  for  the  monitoring  of  pilot  behaviour,  which  is  described  in  more  detail  in 
(Stutz  &  Onken,  2001).  These  modules  establish  the  cognitive  sub-process  (2)  of  situation  interpretation 
within  the  cognitive  loop. 

On  the  basis  of  situation  knowledge,  possible  conflicts  ahead  can  be  identified  (e.g.  threats,  weather)  by  the 
Flight  Situation  and  Threat  Interpreter  (FTI).  Here  the  impact  on  the  current  flight  is  assessed,  conflicts  are 
detected,  and  resolution  activities  are  initiated.  The  module  FTI  is  the  implementation  of  the  cognitive  sub¬ 
process  (3)  of  goal  activation. 


Figure  9:  Functional  Structure  of  the  Crew  Assistant  Military  Aircraft 
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The  Misson  Planner  (MP)  generates  a  complete  3D/4D  mission  plan  either  on  demand  by  the  crew  or 
autonomously,  where  the  crew  does  not  have  the  resources  to  interact.  The  mission  plan  consists  of  both  IFR- 
and  low-level  flight  segments.  The  module  MP  can  be  seen  as  the  representation  of  the  cognitive  sub-process 
(4)  of  planning  and  decision  making. 

The  interface  between  CAMA  and  the  crew  is  controlled  by  the  module  Dialogue  Manager  (DM).  Speech 
output  is  being  used  for  focusing  the  pilot’s  attention  on  the  important  aspects.  More  complex  information  is 
transmitted  using  graphical  displays.  Information  input  is  realised  utilising  speech  recognition  to  a  large  extent 
(Flemisch  &  Onken,  1998).  The  DM-related  functions  cover  comprehensive  parts  of  the  cognitive  sub¬ 
processes  (1)  data  acquisition  from  the  cockpit  crew  as  part  of  the  machine-external  world  and  (6)  control  and 
execution  of  the  derived  actions  with  respect  to  the  pilots. 

After  successful  module  integration,  CAMA  was  tested  thoroughly  in  two  flight  simulator  campaigns  in 
1997/98  (Schulte  &  Stiitz,  1998).  The  results  of  these  investigations  led  to  further  consolidation  of  the 
implemented  functions. 


4.1.2  Experimental  design 

In  spring  and  fall  2000  CAMA  was  integrated  in  the  experimental  cockpit  of  ATT  AS,  the  Advanced 
Technology  Testing  Aircraft  System  of  the  DLR,  based  on  a  VFW  614  aircraft,  in  order  to  be  evaluated  in 
flight  trials.  The  following  paragraphs  give  an  overview  over  the  experimental  design  prepared  for  the  two 
flight  test  campaigns.  (See  also  Stiitz  &  Schulte,  2000  for  the  results  of  the  first  campaign.) 

Apparatus,  scenario  and  tasks 

The  experimental  cockpit  of  ATT  AS  enables  the  investigation  of  advanced  avionics  functions  under  real  flight 
conditions,  thereby  providing  full  access  to  the  on  board  fly -by -wire  system  and  navigational  means.  Standard 
interfaces  to  the  aircraft  are  given  through  a  side-stick,  a  flight  control  unit  (FCU),  a  radio  management  unit 
(RMU)  as  well  as  levers  for  gear,  flaps  and  spoilers. 

Three  CRTs  were  used  to  depict  CAMA’s  primary  flight  displays,  the  navigation  display  and  a  third 
supplemental  display,  each  equipped  with  touch  screens.  In  order  to  enable  low  level  flying  from  the 
experimental  cockpit,  which  is  located  in  the  passengers  bay  of  the  aircraft.  A  LCD-projector  provided  an 
outside  view  taken  from  a  video  camera  in  the  aircraft’s  cockpit,  enhanced  by  a  3D  head-up  display  overlay. 

Furthermore,  appropriate  installations  were  made  to  enable  verbal  communication  between  CAMA  and  the 
pilot.  Speech  output  was  derived  using  a  DEC  DecTalk  System,  for  speech  recognition  a  SUN  SSI  system  was 
integrated.  The  CAMA  main  modules  were  hosted  by  two  SGI  workstations  interfaced  via  ethemet  to  the  core 
avionics  system  of  ATT  AS.  (See  Figure  10  for  an  overview  of  the  hardware  set-up.)  In  order  to  be  compliant 
with  the  experiments  previously  conducted  in  the  simulator  trial  a  similar  mission  was  prepared  and  agreed 
with  ATC. 


Figure  10  Integration  of  CAMA  hardware  into  A  TTAS 
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Takeoff  was  from  Landsberg  air  base  near  Munich.  An  IFR-transit  flight  via  ATC  routes  led  the  aircraft  to  the 
gaming  area  located  over  the  Black  Forest  area  in  Germanys  south-west,  which  provides  sufficiently 
structured  terrain  for  realistic  tactical  terrain  masking.  Here  low-flying  had  to  be  performed  until  the  target 
point  was  reached  and  a  simulated  drop  procedure  was  earned  out.  The  altitude  flown  during  the  low-level 
segment  was  at  a  minimum  of  1,000  ft  above  ground  due  to  safety  considerations.  After  leaving  the  battle 
zone,  IFR-flight  was  again  performed  back  to  the  home  base.  The  duration  of  the  flight  was  about  1.5  hours. 

During  the  mission  the  pilot  had  to  handle  typical  piloting  tasks.  Changes  in  air  tasking  and  airspace  co¬ 
ordination  order  repeatedly  called  for  re-planning.  In  addition  other  scenario  elements  like  adverse  weather 
conditions  and  surface-to-air  missile  sites  were  simulated. 

Subjects 

During  the  two  CAMA  flight  trial  campaigns  a  total  of  five  German  Air  Force  transport  pilots  from  the  Air 
Transport  Wing,  Landsberg  were  envolved  as  subjects.  They  were  rated  combat  ready  on  C-160  Transall  and 
had  logged  850  and  4200  hours  as  pilot  in  command.  The  second  campaign  was  supported  by  two  test  pilots 
from  the  Technical  and  Airworthiness  Centre  for  Aircraft,  Manching.  During  the  campaigns  eleven  missions 
were  conducted.  Only  one  mission  was  corrupted  by  serious  hardware  problems,  so  that  this  mission  could  not 
be  considered  for  evaluation. 


4.1.3  Evaluation  results 

The  evaluation  was  performed  mainly  to  demonstrate  the  technical  feasibility  of  a  complex  crew  assistant 
system  being  operated  under  real-world  conditions.  In  order  to  assess  the  pilot’s  overall  acceptance  of  the 
approach  and  benefits  offered  by  the  cockpit  assistant  system,  a  de-briefmg  session  concluded  the 
experiments,  where  the  pilots  were  asked  to  fill  out  subjective  ratings. 

The  methodological  approach  to  the  subjective  evaluation  of  CAMA  followed  a  certain  scheme  of  questions, 
which  has  been  derived  from  the  considerations  mentioned  below.  Rouse  (1991)  proposes  the  following 
aspects  for  the  evaluation  of  man -machine-systems: 

•  Compatibility :  Is  the  system  adapted  to  the  operators  capabilities  and  limitations? 

•  Intelligibility:  Does  the  organisation  and  the  contents  of  the  man-machine-dialogue  provide  a  meaningful 
communication? 

•  Effectiveness :  To  what  extent  does  the  system  enhance  task-performance? 

According  to  Johannsen  (1993),  meeting  these  three  criteria  is  a  prerequisite  to  reach  a  high  degree  of 
acceptance  for  the  man-machine-system.  Another  important  aspect  for  the  evaluation  of  a  crew  assistant 
system  is  the  term  of  situation  awareness.  Endsley  (1988)  defines  situation  awareness  as  follows: 

“Situation  awareness  is  the  perception  of  the  elements  in  the  environment  ...  ,  the 
comprehension  of  their  meaning,  and  the  projection  of  their  status  in  the  near  future.” 

Therefore,  the  questionnaires  were  structured  according  to  the  following  aspects: 

•  In  order  to  evaluate  the  situational  awareness  aspects,  the  considered  situation  space  was  classified  and 
structured  into  classes  and  sub-classes  of  relevant  situational  elements.  Again,  with  regard  to  these  classes 
the  pilots  had  to  indicate  whether  they  had  all  the  situational  element-related  information  at  their  disposal 
whenever  needed. 

•  The  effectiveness  and  the  benefit  provided  by  the  functions  were  evaluated  by  listing  all  relevant  tasks  and 
sub-tasks  throughout  the  mission.  The  subjects  had  to  comment  on  the  quality  of  assistance  provided  by 
CAMA  with  respect  to  these  tasks. 

•  To  evaluate  the  degree  of  acceptance,  the  pilots  had  to  refer  to  a  list  of  statements  characterising  the 
system  behaviour  and  handling  features. 
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All  ratings  were  given  on  a  scale  of  1  to  7,  where  1  represents  the  best  evaluation  and  7  marks  the  negative 
end  of  the  scale.  The  markers  depicted  on  the  following  ratings  stand  for: 


Simulator  trials  1,  2,  3: 


Median  values  from  10  different  pilots  after  having  flown  the 
mission  for  the  first  (1),  the  second  (2)  and  the  third  time  (3) 


Second  flight  trial: 


Exact  value  from  four  individual  ratings  (1  -  4)  after  having 
flown  live-missions 


The  following  aspects  were  assessed: 


Situation  awareness 

In  order  to  evaluate  the  situational  awareness  aspects,  the  situation  space  considered  was  classified  and 
structured  into  classes  and  sub-classes  of  relevant  situational  elements.  The  pilots  had  to  give  their  impression 
of  the  specific  awareness  qualities.  Figure  1 1  shows  some  selected  results. 

"How  would  you  judge  your  overall  awareness  concerning ...  during  flight?" 


a)  "aircraft  attitude" 

good 

bad 

b)  "terrain  relief ' 

good 

bad 

mt/\  'M,  ! 

M. 

1  2  3 

4  7 

2  3 

4  7 

c)  "threat  along  low-level  trajectory" 

good 

bad 

d)  f 1 being  aware  of  my  own  faults " 
always 

never 

w  ; 

1  2  3 

4  7 

1  2  3 

4  7 

Figure  11:  Ratings  concerning  situational  awareness 

Results  1  la-d  basically  confirm  the  findings  from  the  simulator  trial.  The  rather  critical  rating  of  one  pilot  in 
Figure  11a  and  b  indicate  some  problems  with  the  enhanced  vision  terrain  depiction  on  the  HUD  in 
combination  with  the  unusual  out-of-the-window  view  provided  through  a  video  set-up  (see  above). 


Assistance  Quality 

Here  the  pilots  were  asked  to  rate  quality  aspects  of  the  assistance  provided  by  CAMA  while  conducting 
typical  mission  tasks. 


"How  did  CAMA  assist  you  in  ... " 

a)  "complying  with  mission  constraints  b)  "planning  the  low  level  routing?” 
( ACO/ATO )?" 


good  _  bad  good  _ _ bad 


] 

1  2  2 

5  4  ' 

7  1  2  3  4 

7 

c)  "updating  flight  plan  due  to  ATC 

d)  "performing  low-level  flight  guidance 

clearances/  orders?" 

&  navigation?" 

good 

bad 

good 

bad 

_ i _ A _ _ J 

TIAT!  A 

1  2  3  4  7  1  2  3  4  7 


Figure  12:  Ratings  concerning  assistance  quality 
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Figure  12  a&b  also  confirm  the  results  found  in  the  simulator  trails.  Rating  c  can  be  explained  by  the  shared 
responsibility  for  ATC-comm unication  between  experimental  and  safety  pilots.  Slight  deficits  in  the  ratings  as 
compared  to  the  simulator  trials  are  certainly  induced  by  a  lack  in  familiarisation  with  the  experimental  set-up. 

Handling  Quality  and  Acceptance 

The  acceptance  was  determined  by  subjectively  evaluating  the  behaviour  of  the  assistant  system  as  well  as  its 
handling  qualities.  For  example  the  appearance  of  CAMA’s  monitoring  and  advice  functionality  as  well  as  the 
general  handling  had  to  be  rated. 


a)  " The  advisories  and  warnings  given  by  CAMA  were  ... 

necessary 
useful 
sensible 
apropriate 

1  2  3  4  7 


7®;  Af 

1  IS 

f'Av'f  A\  /F 

L _ _ _ 

not  necessary 
useless 
nonsense 
inapropriate 


b)  ” The  handling  of  CAMA  was  " 

easy  difficult 


1  2  3  4  7 


Figure  13  Ratings  concerning  handling  quality  and  acceptance 

Deficiencies  in  the  advisory  behaviour  (Figure  13  a),  which  resulted  in  erroneous  warnings,  could  be  traced 
back  to  insufficient  adaptation  of  CAMA’s  intern  user  model  to  ATTAS  procedures.  Overall  handling  quality, 
though,  proved  to  be  unchanged  compared  to  the  simulator  runs  (Figure  13  b). 


Summary 

CAMA  and  its  various  assistant  functions  have  previously  been  evaluated  during  two  experimental  campaigns 
utilizing  a  flight  simulator.  In  spring  and  fall  2000  CAMA  was  tested  for  the  first  time  in  in-flight  trials  in  two 
separate  campaigns.  The  CAMA  system  was  installed  on-board  the  ATTAS  experimental  aircraft.  Special 
provisions,  e.g.  a  video  outside  view  had  to  be  made  to  enable  low-flying  from  the  experimental  cockpit 
located  in  the  passengers  bay. 

Professional  military  pilots  performed  a  total  of  eleven  transport  missions.  In  the  de-briefing  sessions  the 
pilots  gave  generally  positive  ratings  to  the  system,  its  contribution  to  situation  awareness,  the  quality  of  the 
assistant  functions,  and  the  degree  of  acceptance  of  such  an  electronic  crew  member.  A  few  less  positive 
ratings  had  to  be  accepted  in  certain  areas.  These  can  mainly  be  attributed  to  shortcomings  in  the 
familiarisation  of  the  subjects  with  the  experimental  environment  as  well  as  the  limits  of  the  adaptation  of 
CAMA’s  functionality  to  the  ATTAS  aircraft.  The  promising  results  gained  in  the  simulator  trials  could 
nevertheless  be  confirmed. 


4.2  Tactical  Mission  Management  for  Attack  Aircraft 

In  order  to  prove  the  cognitive  system  approach  in  the  field  of  military  fighter  aircraft,  the  Tactical  Mission 
Management  System  has  been  implemented  as  a  functional  prototype  and  integrated  in  the  flight  and  scenario 
simulation  environment  at  ESG.  It  has  undergone  an  experimental  evaluation  with  operational  personnel  in 
spring  2001.  The  following  sections  give  details  on  the  experimental  design  and  the  evaluation  results. 

4.2.1  Functional  architecture 

In  order  to  derive  a  functional  breakdown  of  the  TMM  the  generic  processing  steps  are  translated  into  specific 
functions  (see  Figure  14  and  Figure  5  as  a  reference).  On  the  state -oriented  layer  situational  parameters  such 
as  aircraft  sensor  and  system  signals,  data-link  information  such  as  tactical  elements  and  mission  order,  and 
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on-board  database  entries  e.g.  terrain  elevation  data  are  to  be  considered.  These  data  are  analysed  in  order  to 
identify  context-specific  features  concerning  pilot  behaviour,  aircraft  movement  and  the  external  tactical 
situation.  The  monitoring  of  the  threat  accumulation  along  the  planned  trajectory  is  a  typical  example  for  a 
context-spanning  analysis. 


Figure  14:  Functions  of  machine  problem  solving  for  mission  management 

The  determination  of  the  current  phase  of  flight  is  essential  for  the  model-based  prediction  of  expected  crew 
actions  required  to  cope  with  the  mission  constraints.  It  considers  all  relevant  task  domains  such  as  flight 
guidance  along  the  pre-planned  track,  systems  operation,  weapon  deployment,  as  the  human  operator  does.  It 
computes  the  current  tasks  and  task  parameters  relevant  for  the  crew  in  the  present  situational  context  (see 
Figure  8). 

The  conflict  detection  and  resolution  function  builds  up  a  hierarchy  of  general  goals  to  be  followed  throughout 
the  mission,  such  as  flight  safety,  combat  survival  and  mission  accomplishment  (see  Figure  7).  Utilising  the 
results  of  the  tactical  situation  interpretation  and  the  monitoring  of  the  flight  situation-dependent  tasks,  the 
system  figures  out  violations  of  these  goals.  After  negotiation  with  the  pilot,  a  proposal  how  to  resolve  the 
conflict  will  be  passed  to  appropriate  machine  agents  for  conflict  resolution.  Implementing  this  human -like 
goal-task-model  ensures  machine  problem  solving  strategies  which  are  easy  to  anticipate  for  the  pilot. 

Planning  is  the  most  important  conflict  solving  agent  activity.  The  tactical  mission  management  system  offers 
a  fully  autonomous  mission  and  route  planning  capability  to  the  crew,  including  terminal  operations  planning, 
transit  flight  planning,  tactical  low-level  flight  trajectory  planning  and  the  use  of  attack  procedure  templates. 
Feedback  is  obtained  in  two  ways;  externally,  due  to  manipulatory  action  of  the  crew  and,  thereby,  alteration 
of  the  situation;  and  internally  by  the  continuous  re -consideration  of  the  planning  result  in  terms  of  goal 
integrity. 

Finally,  the  Tactical  Mission  Management  System  provides  an  appropriate  man-machine  interface  on  the 
flight  deck,  in  order  to  manage  the  information  flow  and  the  crew  interactions.  The  main  components  are  an 
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advanced  primary  flight  display  utilising  a  perspective  3D  synthetic  vision  symbology,  a  tactical  mission 
management  and  navigation  interface,  and  speech  synthesis. 

4.2.2  Experimental  design 

In  order  to  prove  the  approach,  the 
TMM  has  been  implemented  as  a 
functional  prototype  and  integrated 
in  the  flight  and  scenario 
simulation  environment.  It  has 
undergone  an  experimental 

evaluation  with  operational 
personnel  in  spring  2001.  The 
following  sections  give  details  on 
the  apparatus,  scenario  and  tasks 
as  well  as  the  subjects. 


Apparatus,  scenario  and  tasks 
For  the  evaluation  of  the  Tactical 
Mission  Management  System  a 
comparative  study  was  chosen. 

Two  different  simulator  set-ups 
were  configured,  on  the  one  hand 
representing  the  basic  functions  of 
a  reference  combat  aircraft  cockpit 
(e.g.  Tornado)  and  on  the  other 
hand  demonstrating  the  TMM 
functions  and  displays  (Figure  15). 

Following  the  experimental 
procedures  the  pilots  had  to 
perform  a  dedicated  test  mission  with  each  of  the  cockpit  configurations.  Figure  16  shows  the  phases  of  the 
test  mission  located  in  the  south-west  regions  of  Germany  i.e.  (1)  tactical  transit,  (2)  low-level  ingress,  (3) 
attack,  (4)  low-level  egress  and  (5)  tactical  transit.  During  the  low-level  phase  the  mission  was  supported  by 
computer-generated  units  such  as  SEAD-forces  for  suppression  of  enemy  air  defence  and  AW  ACS.  Using  the 
TMM-configuration  the  aircraft  was  participant  of  a  tactical  data-link  network  providing  data  on  other  partici¬ 
pants  and  surveillance  information.  During  the  mission  the  tactical  situation  (i.e.  hostile  SAM  sites)  was 
supposed  to  change  several  times  forcing  the  pilots  to  react  accordingly  (e.g.  route  adaptation,  re -planning, 

threat  avoidance),  thereby  workload 
being  imposed  on  the  operator. 

Subjects 

The  subjects  were  four  German  Air 
Force  pilots  (partly  flight  instructors) 
from  the  Fighter  Bomber  Wing  34, 
Memmingen  at  an  age  of  30  to  38  years. 
Their  flight  experience  ranged  from  a 
total  of  900  to  3000  flying  hours  on 
Tornado  and  other  NATO  combat 
aircraft.  During  a  one  day  familiarisation 
phase  the  pilots  had  the  opportunity  to 
train  the  handling  of  the  simulator  and 

_  T  _  _  the  interaction  with  the  TMM  before  the 

Figure  16:  Scenario  of  the  test  mission  for  system  evaluation  (est  misskm  had  t0  be  performed. 
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Figure  15:  Cockpit  in  the  TMM  configuration 
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4.2.3  Evaluation  results 

The  main  scope  of  the  evaluation  of  the  Tactical  Mission  Management  System  was  to  account  for 
improvements  in  comparison  to  the  reference  system  in  terms  of  the  following  categories  of  measurements: 

•  Measurement  of  pilot  performance 

•  Evaluation  of  system  performance  of  the  TMM 

•  Evaluation  of  pilot’s  workload 

•  Measurement  of  pilot’s  situation  awareness 

•  Subjective  ratings  concerning  TMM  performance 

The  following  paragraphs  report  on  the  specific  results. 


Pilot’s  performance 

The  investigation  of  the  pilot’s  performance  has  been  conducted  under  the  consideration  of  the  three  abstract 
goals  as  defined  earlier  in  this  paper:  flight  safety,  combat  survival  and  mission  accomplishment. 

With  regard  to  flight  safety ,  the  area  of  low- 
level  flight  guidance  has  been  investigated. 
During  the  experiments  it  could  be  observed 
that  pilots  frequently  took  the  risk  of  dangerous 
ground  and  obstacle  proximity  in  order  to  avoid 
military  threats.  Therefore,  it  was  investigated 
how  often  certain  given  above-ground-level 
minima  were  violated  while  performing  low- 
level  flight.  Table  1  gives  the  results  of  the 
comparison  between  the  reference  system  (REF) 
and  the  Tactical  Mission  Management  System 
(TMM).  The  assessment  of  ground  collisions 
and  the  frequency  of  AGL  minima  violations 
make  clear  that  the  TMM  caused  a  significant 
risk  reduction  by  a  better  ground  separation.  In  a  deeper  investigation  of  the  terrain  following  performance 
(Schubert  &  Schulte,  2001)  it  is  evident  that  the  flown  vertical  profile  becomes  significantly  smoother  (i.e. 
less  vertical  acceleration,  less  variation  in  altitude)  in  the  TMM  configuration.  Thereby,  the  pilot’s  comfort 
level  could  be  increased. 


Table  1:  Violations  of  minima  during  low-level  flight 


REF 

<  Oft  AGL 

<50  ft  AGL 

<  150  ft  AGL 

Pilot  1 

1 

5 

39 

Pilot  2 

0 

0 

17 

Pilot  3 

1 

3 

19 

Pilot  4 

1 

9 

36 

TMM 

<  Oft  AGL 

<50  ft  AGL 

<  150  ft  AGL 

Pilot  1 

0 

0 

2  i 

Pilot  2 

0 

0 

5 

Pilot  3 

0 

0 

5 

Pilot  4 

0 

1 

23 

An  important  feature  of  the  TMM 
is  the  ability  of  situation- 
dependent  in-flight  re-planning  of 
the  mission  plan  for  threat 
reduction.  In  order  to  quantify  the 
effect  of  this  assistance  function  in 
terms  of  combat  survival  the  mean 
threat  exposure  has  been  evaluated 
along  the  flown  trajectories 
(Figure  17).  Obviously,  a  massive 
reduction  effect  on  the  threat 
exposure  could  be  noted  by  use  of 
the  TMM.  It  should  be  emphasised 
that  the  improvement  of  threat 
avoidance  with  the  TMM  could  be 
achieved  in  combination  with  a 
much  better  ground  separation  (see 
Table  1). 


Pilot  1  Pilot  2  Pilot  3  Pilot  4 


□  REF 

□  TMM 


Figure  1 7:  Mean  threat  exposure  in  comparison 
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Table  2:  Global  criteria  of  flight  safety  and  mission  accomplishment 


Pilot  1 

Pilot  2 

Pilot  3 

Pilot  4 

REF 

TM 

REF 

TM 

REF 

TM 

REF 

TM 

M 

M 

M 

M 

SAM  shots  [#1 

- 

- 

- 

- 

1 

- 

1 

- 

A  TOT  [s] 

L  8.0 

E  0.4 

E  0.3 

E  2.1 

E  2.3 

E  0.2 

L  3.8 

L  2.1 

Target  hit 

OK 

OK 

OK 

OK 

OK 

OK 

OK 

OK 

Destination  reached 

OK 

OK 

OK 

OK 

OK 

OK 

no 

(OK) 

Fuel  on  Board  @  Touch  down  [%] 

4.4 

14.3 

9.1 

13.8 

4.3 

12.5 

0.0 

8.8 

ACO  violations 

5 

- 

5 

- 

5 

- 

8 

- 

Mission  accomplished 

no 

OK 

OK 

OK 

no 

OK 

no 

(OK) 

Table  2  provides  a  collection  of  criteria  concerning  mission  accomplishment.  The  results  make  evident  that  the 
pilots  performed  notedly  better  with  the  TMM  than  without.  Due  to  a  better  threat  avoidance  with  the  TMM 
the  SAM  shots  could  be  reduced.  Performance  criteria  such  as  meeting  the  Time-over-Target  (TOT),  hitting 
the  target  or  reaching  the  destination  could  be  fulfilled  by  all  pilots  quite  well.  Only  pilot  4  did  not  reach  the 
destination  with  the  reference  system  due  to  a  flame-out  condition.  In  general  it  was  found  that  fuel 
consumption  could  be  decreased  significantly  with  the  TMM  during  the  mission.  Another  observation  was 
made  concerning  the  number  of  violations  of  the  Airspace  Co-ordination  Order  (ACO)  routing,  which  could 
be  totally  eliminated  by  the  use  of  the  TMM.  So,  the  risk  of  being  hit  by  friendly  fire  was  minimised. 


□  %km 
11% 


System  performance 
One  of  the  most  important  features 
of  the  TMM  is  the  pilot  assistance  in 
optimising  a  threat  minimal  route 
under  a  dynamically  changing 
hostile  threat  theatre.  Figure  18 
shows  the  total  and  mean  threat 
exposure  computed  with  an 
underlying  worst-case  scenario. 

Comparing  the  threat  exposure  of  a 
direct  routing  (1st  column)  with  the 
result  of  the  low-level  route  planner 
of  the  TMM  (3rd  column)  makes  the 
advantage  obvious.  The  total  threat 
accumulation  could  be  decreased 
from  about  8500  to  5700  %km.  Due 
to  the  longer  flight  trajectory,  the 
effect  on  the  relative  threat  exposure 
is  even  more  noticeable  (55  down  to 

30%).  The  columns  2  and  4  in  Figure  18  show  the  threat  values  of  the  actual  flown  trajectories,  again  under 
consideration  of  a  worst-case  scenario.  It  is  obvious  that  the  co-operation  between  the  system  and  the  pilots 
yields  another  improvement  in  terms  of  threat  avoidance  due  to  synergetic  effects. 
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Figure  18:  Threat  exposure  reduction 


Workload  and  situation  awareness 

The  TMM  was  designed  to  reduce  the  operator’s  workload  by  providing  functions  to  support  a  better  situation 
awareness  and  particular  automation  functions.  During  the  experiments  measurements  of  situation  awareness 
and  workload  were  conducted.  Therefore,  the  experiment  was  stopped  at  dedicated  points  of  time  in  order  to 
perform  the  NASA  Task  Load  Index  (TLX)  and  the  Situation  Awareness  Global  Assessment  Technique 
(SAGAT)  (Endsley,  1988). 
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Figure  19:  NASA  TLX  (left)  and  SAG  AT  (right)  results  over  mission  phases 

The  evaluations  were  conducted  four  times  each  experimental  run  (reference  system  and  TMM).  The 
measuring  points  were  the  task  situations  1  (Transit  Ingress),  2  (Low-level  Ingress),  4  (Low-level  Egress)  and 
5  (Transit  Egress)  according  to  Figure  16.  Figure  19  shows  the  results  of  the  assessments  averaged  over  the 
four  subjects.  Concerning  the  NASA  TLX  rating  (see  Figure  19,  left)  it  was  found  that  the  overall  workload 
could  be  reduced  massively  by  use  of  the  TMM  with  an  expected  slight  increase  of  workload  during  the  low- 
level  phases  of  the  mission.  The  situation  awareness  assessment  was  based  upon  the  evaluation  of  a  total  of  26 
multiple-choice  questions  concerning  situational  features.  Figure  19  (right)  shows  the  weighted  results.  An 
increase  in  situation  awareness  of  about  10  to  15%,  in  particular  during  the  early  mission  phases,  can  be 
observed. 


Subjective  ratings 

For  a  further  evaluation  of  the  system 
performance  the  experimental  subjects  had 
to  give  subjective  ratings  by  the  use  of 
questionnaires  in  a  de-briefing  session  at 
the  end  of  the  two  day  evaluation  period. 
The  main  aspects  of  the  survey  were  the 
evaluation  of 

•  system  performance  of  the  TMM, 

•  acceptance  of  the  TMM  by  the  user, 
and 

•  overall  assessment. 


good  poor 

situation  gathering:  ACO 
situation  gathering:  Route 
situation  gathering:  Timing 
situation  gathering:  System 
Assessment  SAM-Threat 
Planning  according  to  ATO 
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Mission  Re-planning 
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The  rating  scales  covered  a  range  of 
values  from  1  to  7  and  were  each  labelled 
by  a  pair  of  antithetic  terms  (e.g.  good  - 
bad;  agree  -  disagree).  The  following 
paragraphs  report  on  some  selected  results. 

Figure  20  shows  the  pilots’  evaluations  of 
the  quality  and  performance  of  the 
assistance  functions  offered  by  the  TMM. 
The  overall  assessment  can  be  regarded  as 
very  positive.  Although,  there  can  be 
identified  some  minor  objections  caused 
by  unfamiliar  display  of  timing  and  system 
information,  insufficient  training  with  the 
system  and  some  shortcomings  in  pilot’s 
behaviour  modelling.  Despite  these  (easy 


Figure  20:  Evaluation  of  assistance  quality 
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Figure  21:  Overall  assessment  of  the  TMM 
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to  remove)  deficits  the  assessment  of  the  implemented  prototype  was  almost  optimal. 

Figure  21  shows  some  selected  but  representative  results  of  the  overall  assessment  given  by  the  pilots.  The 
subjects  fully  agreed  with  the  hypothesis  that  the  TMM  provides  a  better  big  picture  in  terms  of  global 
situation  awareness.  TMM  is  qualified  to  increase  mission  efficiency  according  to  the  pilots.  The  operators 
regarded  the  presented  technology  of  Tactical  Mission  Management  and  crew  assistance  to  be  absolutely 
necessary,  suited  and  adequate. 

5  CONCLUSIONS 

After  having  gathered  years  of  experience  in  cognitive  flight-deck  automation  and  crew  assistance,  the  time 
has  come  to  gain  the  results  of  exceedingly  successful  efforts  in  the  field  of  military  transport  as  well  as 
combat  aircraft.  The  functional  breakdown  of  the  Crew  Assistant  Military  Aircraft  (CAMA),  representing  the 
air-transport  domain,  and  on  the  other  hand  the  Tactical  Mission  Management  System ,  being  a  cockpit 
automation  system  for  combat  aircraft,  has  been  derived  from  a  model  of  human  information  processing,  in 
order  to  approach  a  co-operative  automation  principle.  Laboratory  prototype  systems  have  been  evaluated 
with  Air  Force  pilots  in  simulator  and  flight  trials.  Besides  the  fact  that  the  approach  was  very  well 
appreciated  by  the  pilots,  objective  measures  evidence  a  significant  increase  in  performance  in  terms  of  threat 
avoidance  and  mission  efficiency.  These  results  could  be  achieved  in  conjunction  with  a  noticeable  reduction 
of  the  terrain  collision  risk  and  the  operator’s  workload.  Therefore,  the  system  approach  is  highly 
recommended  for  application  in  the  advancement  and  automation  of  future  aircraft  as  well  as  other  military 
and  non-military  application  domains,  where  ever  operator  situation  awareness  and  decision-making  is  crucial. 
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Summary 

As  advanced  crew  support  technologies  will  be  available  more  and  more  in  future  military  aircraft,  it  is  necessary 
to  have  a  good  understanding  of  the  possibilities  in  this  area,  taking  into  account  operational  demands,  technical 
possibilities,  human  factors,  evaluation,  and  validation  aspects.  A  Crew  Assistant  (CA)  is  a  decision  support 
system  for  air  crew,  designed  to  improve  mission  effectiveness  and  redistribute  crew  workload  in  such  a  way  that 
the  crew  can  concentrate  on  its  prime  tasks. 

The  POWER  (Pilot  Oriented  Workload  Evaluation  and  Redistribution)  project  is  a  Netherlands  National 
Technology  Project.  The  project  is  aimed  at  demonstrating  a  generic  CA  environment  and  individual  tactical 
decision  support  tools  to  military  pilots  in  a  simulated  environment,  the  NSF  (National  Simulation  Facility),  a  six- 
degrees  of  freedom  cockpit  in  a  visual  dome. 

The  project  is  a  technology  demonstration  project  to  show  new  CA  features.  An  advanced  software  architecture 
has  been  set  up,  based  on  multi-agent  technology,  where  software  ’’agents”  co-operate  in  sharing  information  and 
using  resources  on  an  as-needed  basis.  Each  agent  is  an  autonomous  piece  of  software  that  is  able  to  anticipate 
courses  of  action  and  performs  its  function  pro-actively. 

Several  prototypes  of  crew  assistant  agents  have  been  developed  and  integrated  in  order  to  facilitate  a  CA 
demonstrator 

A  large-scale  experiment  with  operational  pilots  from  the  Royal  Netherlands  Air  Force  (RNL  AF)  has  been  earned 
out  to  demonstrate  the  effects  of  CA  technology  as  decision  support,  to  validate  tools,  and  to  measure  the  effects 
of  on-board  decision  support  in  enhancing  pilot  situational  awareness. 

This  paper  describes  the  demonstration  CA  environment  and  provides  insight  into  the  different  CA  components. 
Part  one  describes  the  environment  as  a  generic  CA  architecture  that  can  be  installed  on  a  simple  work  station  as 
well  as  in  a  full-scale  simulation  environment.  The  second  part  of  this  paper  describes  the  aforementioned 
experiment,  where  NCMM  and  the  contents  of  the  experiment  will  be  detailed. 


Paper  presented  at  the  RTO  SCI  Lecture  Series  on  “Tactical  Decision  Aids  and  Situational  Awareness  ”,  held  in  Amsterdam, 
The  Netherlands,  1-2  November  2001;  Sofia,  Bulgaria,  8-9  November  2001;  Madrid,  Spain,  12-13  November  2001; 
Maryland,  United  States,  19-20  November  2001,  and  published  in  RTO-EN-019. 


5-2 


PART  I:  A  CREW  ASSISTANT  ARCHITECTURE 

1  Introduction 

Software  and  human  factors  are  becoming  major  part  of  current  and  future  on-board  avionics.  Current 
developments  in  software  engineering  and  advanced  information  processing  techniques  enable  complex  crew 
assistant  applications,  especially  support  of  the  aircraft  crew  in  carrying  out  primary  and  secondary  tasks  is  more 
and  more  provided  by  electronic  systems.  Application  of  crew  assistants  in  fighter  aircraft  is  a  challenging  task, 
both  for  the  research  and  development  of  the  software,  as  for  the  human  factors  aspects  concerning  its  optimal  use 
and  trust  of  the  pilot  in  the  system. 

The  POWER  (Pilot  Oriented  Workload  Evaluation  and  Redistribution)  project,  is  a  Netherlands  National 
Technology  Project.  The  project  is  aimed  at  demonstrating  a  generic  Crew  Assistant  (CA)  environment  and 
individual  tactical  decision  support  tools  to  pilots  in  a  simulated  environment,  the  NSF  (National  Simulation 
Facility),  a  six-degrees  of  freedom  F-16  cockpit  in  a  visual  dome.  This  demonstrator  has  been  the  focus  of  the 
project. 

Current  progress  in  advanced  information  processing,  new  advances  in  human  factors,  and  the  possibility  to 
validate  new  avionics  so  that  pilots  leam  to  trust  the  system,  form  the  basis  for  this  paper  in  order  to  support 
RNL  AF  in  the  acquisition  and  usage  of  new  aircraft.  In  this  paper,  a  generic  simulation  environment  is  discussed, 
that  enables  different  levels  of  crew  assistant  demonstration  and  experimentation.  The  environment  proposed  is 
based  on  multi-agent  technology,  where  a  generic  Crew  Assistant  environment  can  be  plugged  onto  a  "simple" 
scenario  generator  on  a  work  station  or  to  a  full-scale  flight  simulation  facility,  like  the  NSF. 

Several  prototypes  of  CA  agents  have  been  developed  and  integrated  in  order  to  facilitate  the  CA  demonstrator: 

•  A  profile  recognition  agent  takes  input  from  different  sensors  and  recognises  profiles  in  a  data  fusion 
assembled  picture.  The  agent  reasons  with  uncertainty  in  the  observation  and  uses  Bayesian  Belief  Networks 
to  model  the  profile  and  sensor’s  characteristics. 

•  A  manoeuvre  prediction  agent  assesses  an  opponent’s  manoeuvre  and  predicts  the  patters  that  will  be  flown 
when  in  air-to-air  combat.  This  tool  is  based  on  case-based  reasoning  technology. 

•  The  NCMM  (NLR  counter  Measure  Manager)  agent  advises  counter  measures  against  threats,  e.g.  SAM- 
sites.  The  tool  is  based  on  expert  system  technology. 

•  An  HMI  is  designed  with  an  NLR  built  tool,  NADDES  (NLR  Avionics  Display  Development  and  Evaluation 
System),  for  increasing  pilot  situation  awareness. 

This  paper  will  provide  an  overview  of  the  POWER  project.  It  has  been  set  up  in  two  parts,  where  the  first  part 
describes  the  generic  CA  architecture  and  part  two  describes  a  large  scale  experiment  that  has  been  earned  out. 
Chapter  two  will  introduce  crew  assistant  technology  and  chapter  three  will  describe  the  software  architecture, 
which  is  based  on  multi-agent  technology.  Chapter  four  describes  the  technical  aspects  of  the  simulation 
environment  the  experiment.  Chapter  five  describes  the  setting  and  results  of  this  experiment. 

2  Crew  Assistant  Technology 

Fighter  pilot's  workload  is  rapidly  increasing.  Modem  military  operations  take  place  in  a  complex  environment  to 
accomplish  a  spectrum  of  missions.  The  most  important  factor  in  the  increase  in  workload  concerns  the 
operational  environment  of  current  fighter  aircraft: 

•  the  increase  of  the  complexity  of  the  military  environment  in  general  (e.g.  combined  joint  task  forces,  peace 
keeping/peace  enforcement). 

•  the  increase  of  the  complexity  of  the  types  of  missions  to  be  flown. 

•  the  increase  in  the  number  of  different  (kinds  of)  threats. 
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Another  factor  is  the  technological  developments  of  fighter  aircraft.  The  increase  in  aircraft  speed  and  aircraft 
types  and  on-board  systems  causes  the  aircraft  itself  to  become  much  more  difficult  to  manage,  putting  more 
pressure  on  the  crew.  During  high-stress  situations,  the  crew  can  get  overloaded  with  information,  while  it  has  to 
perform  a  multitude  of  actions.  Figure  1  illustrates  the  possible  information  overload  and  decision  making 
process. 


Figure  1.  The  information  requirements  of  the  crew  [ Yannone ,  1985]. 

A  Crew  Assistant  is  an  on-board  decision  support  system  that  supports  the  crew  in  performing  its  mission.  It  aims 
at  improving  mission  effectiveness,  flight  safety,  and/or  survivability  by  providing  the  crew  with  concise  and 
relevant  information,  depending  on  the  mission  phase,  thus  enabling  the  crew  to  concentrate  on  mission  decisions 
and  make  more  effective  decisions.  [Urlings  1995].  Crew  Assistants  are  decision  support  systems  for  air  crew, 
designed  to  improve  mission  effectiveness  and  redistribute  crew  workload  in  such  a  way  that  the  crew  can 
concentrate  on  its  prime  tasks.  Ideally,  a  CA  assists  a  pilot,  or  other  crew  members,  by  providing  the  following 
kind  of  functions: 

•  Acquire  the  necessary  information  and  merge  the  input  from  different  sensor  and  information  systems  into 
one  timely  and  consistent  view  of  the  current  situation  (the  status  of  different  on-board  systems,  the  situation 
outside,  etc.). 

•  Process  the  merged  information  to  give  advice  (weapons  selection,  route  planning,  tactics  evaluation,  fuel 
management,  etc.). 

•  Perform  tasks  autonomously  when  so  instructed  by  the  pilot  or  another  crew  member  (autopilot,  target 
tracking,  systems  monitoring,  etc.). 

As  advanced  crew  support  technologies  will  be  available  more  and  more  in  future  (military)  aircraft,  it  is 
necessary  to  have  a  good  understanding  of  the  possibilities  in  this  area,  taking  into  account  operational  demands, 
technical  possibilities,  human  factors,  evaluation,  and  validation  aspects. 
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The  current  state-of-the-art  in  advanced  information  processing  now  makes  intelligent  pilot  assistant  through  a 
CA  technically  feasible,  including  real-time  on-board  data  acquisition,  fusion,  and  processing.  Advanced 
information  processing  techniques,  like  expert  systems  and  neural  networks,  and  methods  for  constraint 
reasoning,  case  based  reasoning  and  reasoning  with  uncertain  and  incomplete  information,  can  be  used  in  the 
military  cockpit.  Research  in  Human  Factors  technology  aspects,  will  make  the  techniques  available  to  the  human 
operator.  For  an  overview  of  systems  and  technologies,  see  e.g.  [Boers  1999],  [Van  Gerwen  2000-1], 
[Verhoeven  2000]. 

In  the  course  of  the  years,  several  CA  programs  have  been  set  up.  The  Crew  Assistant  Military  Aircraft  (CAMA) 
is  a  German  DoD  programme  that  investigates  the  use  of  an  intelligent  electronic  crew  member  in  the  military 
transport  application  [Strohal  1997],  [Frey  1999].  Within  the  programme,  flight  simulator  trials  have  been 
performed  in  the  simulator  of  the  University  of  the  German  Armed  Forces.  Flight  experiments  are  scheduled  in  an 
in-flight  simulator.  Hands-on  experience  and  feed  back  from  pilots  is  a  factor  in  the  development  of  systems  that 
must  be  gained.  One  project  that  takes  a  human  factors  perspective  on  CAs  is  the  Cognitive  Cockpit  (COGPIT) 
project,  that  has  been  set  up  by  the  UK  Ministry  of  Defence  in  conjunction  with  the  Defence  Evaluation  Research 
Agency  (DERA).  This  project  seeks  to  develop  a  theoretically  grounded,  human -centred  approach  for  guiding  a 
principled  development  of  intelligent  pilot  aiding  concepts  for  cockpit  automation  [Taylor  1998],  [Taylor  2000].  It 
researches  the  cognitive  engineering  aspects  of  the  pilots  to  couple  knowledge-based  systems  for  situation 
assessment  and  decision  support  with  conceptsand  technologies  for  adaptive  automation  and  cockpit  adaptive 
interfaces.  Other  CA  projects  are  Pilot’s  Associate  (PA)  [Holmes  1991],  Rotorcraft  Pilot’s  Associate  (RPA) 
[Collucci  1995],  CoPilote  Electronique,  Cockpit  Assistant  System  (CASSY)  [Onken  1997],  and  the  Electronic 
Copilot  (ECOP)  [Stein  1987]. 

These  programs  take  either  a  technological  or  a  human  centred  focus  on  CAs.  We  believe  that  for  a  good 
integration  of  CA  technology  in  the  cockpit,  the  relationship  between  the  fields  of  Advanced  Information 
Processing  ( A I P )  and  Human  Factors  (HF)  should  be  exploited  further.  The  POWER  project  brings  together  the 
fields  of  military  operations  research,  advanced  information  processing  and  human  factors.  It  combines 
techniques  from  artificial  intelligence  and  human  machine  interaction  in  such  a  way  that  pilots  are  supported  with 
advanced  crew  assistants,  wherewith  the  information  is  provided  to  them  in  a  sense  that  a  fused  picture  of  the 
world  emerges.  Apart  from  the  technological  aspects,  the  POWER  project  has  strong  roots  in  the  analysis  of 
requirements  and  needs  from  operational  fighter  pilots.  The  project  will  help  decision  makers  in  technology 
assessments  for  acquiring  new  aircraft  and  equipment  and  manufacturers  in  making  strategic  decisions  on 
technological  programmes. 

In  many  cases,  information  will  be  time  dependent,  inaccurate  and  incomplete.  The  use  of  uncertain  and 
incomplete  information  should  be  further  investigated  and  must  be  considered  in  the  design  of  the  Human 
Machine  Interface  (HMI).  The  project  therefore,  focuses  on  the  following  challenges: 

•  Provide  a  flexible  demonstration  environment  for  crew  assistant  technology,  based  on  operational  demands 
from  fighter  pilots. 

•  Provide  examples  of  crew  assistant  technology,  based  on  new  advanced  AIP  and  HF  aspects. 

•  Provide  insight  in  new  real-time  reasoning  techniques  for  reasoning  with  uncertainty. 

•  Provide  a  quantitative  scientific  base  that  proves  the  benefit  and  user  acceptance  of  CA  technology. 

•  Perform  experiments  with  the  demonstration  environment  that  lays  a  basis  for  further  work  in  the  field  of 
introducing  CA  technology  to  the  RNLAF. 

3  The  Architecture 

Until  recently,  work  in  on-board  automation  focussed  on  the  introduction  of  single  self-supporting  functions.  The 
advantage  of  this  is  high  reliability  in  case  of  single  failures,  where  other  independent  systems  take  over.  Instead 
of  relying  on  the  information  that  one  sensor  provides,  CA  decision  support  functions  in  essence  focus  on  the 


5-5 


integration  of  information  and  the  provision  of  a  complex  and  fused  picture  to  the  pilot.  This  creates  new  demands 
for  the  software  and  hardware  architecture  and  the  human  factors  aspects. 

Decision  support  functions  are  concerned  with  data  acquisition,  fusion,  and  processing  and  use  information  from 
different  sources,  so  that  an  architecture  is  needed,  where  the  system  does  not  rely  on  one  information  source  and 
does  not  contain  any  critical  processing  nodes.  We  believe  that  an  architecture  based  on  agent  technology  can  and 
will  play  an  important  role  in  the  near  future  in  avionics.  Agent-based  architectures  have  been  introduced  for  on¬ 
board  decision  support  systems  in  e.g.  TANDEM  [Barrouil  1999],  the  Cognitive  Cockpit  [Taylor  2000],  and  for 
on  board  multi-sensor  data  fusion  [Bosse  1999]. 

The  strength  of  software  agents  is  that  they  can  be  made  to  interact  with  other  software  agents  or  human  users. 
Agents  are  “small”  autonomous  black  boxes,  which  handle  their  own  clearly  defined  tasks.  A  system  of  agents 
that  co-operates  is  called  a  multi-agent  system.  Agents,  if  well-designed  as  separate  processing  units,  enable 
communication  between  multiple  subsystems,  without  putting  a  strain  on  one  specific  part  of  the  system.  Their 
loose  coupling  provides  a  possibility  to  introduce  new  technology  throughout  the  aircraft's  lifetime,  especially  at 
"end  of  the  line"  functions,  like  weapon  systems,  where  it  is  relatively  easy  to  validate  new  technology. 

This  chapter  will  give  an  overview  of  the  proposed  multi -agent  architecture  and  will  describe  the  most  important 
CA  components  that  have  been  provided  with  the  POWER  project. 

3.1  A  Functional  Architecture  based  on  Multi-Agent  Technology 

The  proposed  architecture  has  been  based  on  the  results  of  earlier  projects,  like  the  EUCLID  (European  Co¬ 
operation  for  the  Long  Term  in  Defence)  Research  and  Technology  project  6.5  [Zuidgeest  1995].  This  NLR-led 
project  on  CAs  for  military  aircraft  started  with  extensive  user  interviews  to  establish  an  inventory  of  operational 
user  problems  and  needs  for  pilots  flying  F-16,  Tornado  and  AM-X.  The  project  came  up  with  a  generic  on-board 
CA  architecture  and  indicated  a  challenge  in  the  application  of  multi-agent  systems  and  knowledge  based  systems. 

The  architecture  that  has  been  set  up  for  the  POWER  project  distinguishes  four  groups  of  functional  agents.  The 
groups  are  (1)  data  and  information  input  agents,  like  sensors  and  the  multi -sensor  data  fusion  agent,  (2)  data 
processing  agents  which  form  the  actual  crew  assistant  functions,  (3)  information  output  agents  mainly  to  the 
pilot,  and  finally,  (4)  the  weapon  agents.  Apart  from  these,  other  agents  perform  functions  for  controlling  and 
monitoring  the  overall  system's  status  and  health.  In  this  paper,  we  will  focus  on  the  functional  part  of  crew 
assistants,  see  figure  2. 

The  four  functional  groups  further  subdivided  in  seven  subgroups  (see  slight  colour  differences  in  figure  2), 
discussed  below. 

Internal  sensor  agents  are  system  components  that  transform  the  raw  input  data  from  the  sensor  hardware  to  an 
understandable  format  for  the  Multi-Sensor  Data  Fusion  (MSDF)  component.  In  our  example,  we  included 
sensors  to  detect  and  distinguish  SAMs  and  to  detect  incoming  missiles. 

•  A  Radar  Warning  Receiver  (RWR)  provides  the  position  of  ground  threats,  including  an  indication  whether 
the  SAM  is  in  search,  track,  or  guidance. 

•  The  Missile  Launch  Warning  (MLW)  is  a  passive  infrared  plume  detector  that  provides  missile  information 
while  its  motor  is  burning. 

•  The  Missile  Approach  Warning  is  an  active  short  range  radar  that  detects  a  missile  body,  usually  in  a  two  to 
three  miles  range. 
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Figure  2.  Functional  architecture  of  on-hoard  agents 

External  sensor  agents  are  components  that  obtain  their  information  from  sensors  or  information  systems  that  are 
physically  located  outside  the  aircraft,  for  example  an  AWACS  or  a  Link-16.  These  sensor  agents  transform  data 
and  information  into  an  understandable  format  for  the  MSDF  agent  or  for  the  CA  agents. 

The  Multi-Sensor  Data  Fusion  agent  combines  the  sensor  information  from  all  internal  and  external  sensors  into  a 
combined  sensor  data  picture.  This  agent  may  perform  complex  situation  assessment  tasks.  In  the  current 
implementation,  this  is  a  fusion  process  that  only  provides  the  information  to  the  CA’s  that  is  really  necessary  for 
the  CAs  to  perform  their  task.  Different  projects  have  already  shown  the  complexity  of  a  multi-sensor  data  fusion 
process  and  have  proposed  architectures  [TA-10],  [Bosse  1999].  The  latter  proposes  an  agent  based  architecture 
for  multi-sensor  data  fusion,  which  shows  the  flexibility  of  agent  systems,  where  agents  can  delegate  tasks  to 
(sub-)agents. 

Crew  Assistant  agents  are  the  intelligent  pilot  support  functions.  The  ones  mentioned  in  figure  2  are  elaborated  in 
the  POWER  project  (based  on  [Zuidgeest  1995]),  however,  the  range  of  pilot  support  functions  is  not  limited  to 
these.  CAs  can  be  further  classified  into  functions  as  information  finding  in  the  fused  sensor  picture  (like  profile 
recognition,  see  section  3.2),  pilot  advice  (like  manoeuvre  prediction,  see  section  3.3,  and  NLR's  Counter  Measure 
Manager,  see  section  3.4  and  chapter  4),  pilot  monitoring,  mission  monitoring,  etc.  Other  classifications  are 
possible,  like  [Barrouil  1999],  [Taylor  2000]. 

Weapon  agents  control  the  weapon  delivery.  In  this  example,  a  number  of  softkill  weapons  to  countermeasure 
ground  threats  is  displayed.  Their  intelligence  for  example  consists  of  providing  the  correct  jamming  against  a 
recognized  threat  or  dispensing  a  complex  pattern  of  chaff  and  flare. 
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The  Display  agent  is  responsible  for  assembling  an  integrated  picture  of  crew  assistant  information  and  for 
prioritizing  information  provision  to  the  pilot.  If  necessary,  it  can  hold  information  that  is  considered  less 
important  at  a  certain  moment  or  less  time  critical,  if  the  pilot  is  assumed  to  get  overloaded  with  information. 
Once  the  situation  is  more  relaxed,  it  can  decide  to  provide  this  information.  An  even  more  intelligent  display 
agent  can  decide  what  information  should  be  provided  on  which  cockpit  display,  or  what  information  should  be 
provided  on  which  part  of  the  cockpit  display  and  automatically  adapt  the  contents  of  the  available  displays  if  at 
(a  certain  part  of)  one  of  the  displays  an  information  overload  is  eminent.  This  technology,  however,  should  be 
introduced  with  care  [Verhoeven  2000]. 

The  Human  Machine  Interface  agent  is  the  actual  cockpit  display  that  provides  the  information  on  the  user 
interface.  It  may  take  inputs  from  the  user. 

For  a  generic  CA  demonstration  environment,  we  require  a  crew  assistant  independently  from  its  operational 
environment.  Obviously,  a  generic  demonstration  environment,  especially  a  complex  one  as  a  crew  assistant 
requires  a  number  of  general  tools,  like  scenario  generation  tools  and  environment  databases.  The  currently 
developed  environment  connects  the  tools  to  ITEMS,  which  provides  a  topographical  scenario  and  an  F-16 
aircraft  model.  This  enables  the  possibility  to  connect  the  crew  assistant  to  e.g.  a  fly  box,  see  figure  3,  to  the 
NLR’s  F-16  mock  up  simulator,  and  to  the  NSF,  see  figure  4. 


Figure  3.  Flybox  and  workstation  configuration  Figure  4.  National  Simulation  Facility 

3.2  Reasoning  with  Uncertainty  for  Profile  Recognition 

Reasoning  with  uncertainty  will  be  an  important  aspect  of  CA  technology.  Even  the  information  from  a  fused 
sensor  picture  will  usually  contain  unclear  elements.  To  investigate  possibilities  of  real-time  reasoning  with 
uncertainty,  a  profile  recognition  agent  was  developed. 

Any  crew  assistant  will  gather  information  from  its  environment,  process  this  information,  and  act  upon  it. 
Probabilistic  methods  for  reasoning  with  uncertainty  have  gained  a  lot  of  interest  in  the  last  few  decades.  The 
introduction  of  Bayesian  Belief  Networks  [Pearl  1988],  [Jensen  1996]  made  practical  application  of  probabilistic 
reasoning  possible.  Most  applications  have  been  targeted  at  decision  support  in  the  medical  domain  where  the 
variables  in  the  model  typically  have  a  few  states  and  no  real-time  constraints  exist.  In  contrast,  the  decision 
support  systems  for  fighter  pilots  have  to  deal  with  many  real-valued  sensor  readings  that  provide  measurements 
every  split  second  and  that  need  to  be  processed  in  real-time  [Van  Gerwen  2000-2]. 
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Suppose  an  aircraft  has  on-board  sensors  to  identify  profiles.  In  this  example,  different  profiles  are  considered  and 
the  aircraft  sensors  can  measure  length,  width,  and  speed  of  the  objects  that  are  sensed.  However,  sensor  accuracy 
depends  on  weather  conditions,  in  particular  visibility,  cloud  coverage,  and  humidity.  This  scenario  is  modelled 
through  a  Bayesian  Belief  Network  as  given  in  figure  5.  Each  of  the  profiles  modelled,  has  an  a-prior  probability 
distribution. 


Figure  5.  Bayesian  Belief  Networks  for  profile  recognition 

The  goal  of  this  research  is  to  look  into  the  possibility  of  creating  an  anytime  algorithm  for  a  Bayesian  Belief 
Network.  Anytime  algorithms  are  algorithms  that  trade  performance  for  time.  As  the  amount  of  time  is  increased, 
an  anytime  algorithm  improves  the  quality  of  the  output.  One  of  the  features  of  anytime  algorithms  is  that  it  can 
provide  intermediate  results  at  any  moment,  so  that  the  available  processing  power  and  time  can  be  regulated. 

Propagating  evidence  in  a  Bayesian  belief  network  can  be  a  very  time-consuming  task  (it  is  NP-hard).  One 
solution  might  be  an  algorithm  that  uses  state-space  abstraction.  We  examined  different  AI  techniques,  like 
Breadth  First,  Breadth  First  Split,  and  Highest  Belief  First.  Figure  6  gives  the  results  of  the  different  methods. 

In  the  CA  agent,  we  have  shown  that  state-space  abstraction  in  combination  with  a  strategy  such  as  Highest  Belief 
First  has  interesting  features  required  for  an  anytime  algorithm.  The  proposed  method  outperforms  simpler 
methods  like  Breadth  First  and  Breadth  First  Split. 
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■  time  (ms) 


Figure  6.  Running  the  BBN  with  different  techniques  to  facilitate  state  space  abstraction 

3.3  Manoeuvre  Prediction  with  Case  Based  Reasoning 

The  proposed  Manoeuvre  Prediction  agent  seeks  to  enhance  the  situation  awareness  of  the  pilot  in  a  dogfight 
situation.  Given  information  on  the  opponent’s  aircraft,  like  position  and  speed,  the  agent  will  try  to  determine  the 
manoeuvre  it  is  performing.  Using  this  information,  the  agent  can  predict  the  future  path  of  the  opponent’s 
aircraft.  This  information  is  then  presented  to  the  pilot,  thus  enhancing  situational  awareness. 

The  rapid  changing  environment  in  which  such  an  application  has  to  operate  called  for  a  strict  real-time  or 
any-time  approach  for  the  Manoeuvre  Prediction  agent.  We  also  wanted  to  exploit  the  fact  that  every  encountered 
dogfight  involves  manoeuvres  that  the  application  could  use  for  future  manoeuvre  prediction. 

It  was  decided,  to  use  the  case  based  reasoning  technique  from  the  field  of  artificial  intelligence  for  implementing 
the  Manoeuvre  Prediction  agent.  The  case  bases  reasoning  technique  will  store  cases  (one  manoeuvre  is  one  case) 
in  a  database  and  provide  means  by  which  a  newly  encountered  case  can  be  compared  to  existing  cases.  The  most 
similar  case  (manoeuvre)  can  be  retrieved  and  used  as  source  to  indicated  the  encountered  case.  The  newly 
encountered  case  will  become  part  of  the  database  for  future  use.  This  way  of  working  is  highly  intuitive  for 
humans,  we  leam  from  experience  and  try  to  recognise  by  comparing  the  current  situation  to  past  experiences. 

The  scientific  challenge  for  implementing  the  Manoeuvre  Prediction  was  to  make  the  case  based  reasoning 
process  work  in  real-time.  We  solved  this  by  ordering  the  cases  by  individual  elements  of  the  manoeuvre  (e.g. 
speed  or  distance  from  own-ship).  Having  an  order  in  the  database  gives  fast  information  where  to  look  for  the 
most  similar  case.  In  the  end  the  search  algorithm  will  quickly  find  the  most  similar  case,  but  more  importantly 
the  algorithm  will  also  perform  well  in  anytime  and  will  produce  a  fairly  similar  case,  once  interrupted. 

Using  the  advanced  software  architecture  of  multi-agent  technology  the  results  from  the  Manoeuvre  Prediction 
application  are  made  available  to  the  other  applications  (agents).  These  agents  can  use  this  information  for  their 
own  purposes.  The  HMI  agent  will  graphically  display  the  information  for  the  situational  awareness  of  the  pilot. 
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The  NCMM  agent  may  use  this  information  to  take  counter  measures  based  on  the  enemy’s  aircraft  predicted 
position.  The  information  may  be  sent  to  agents  of  nearby  ground  or  airborne  forces  for  their  tactical  decision  aid 
or  situational  awareness. 

3.4  NLR’s  Counter  Measure  Manager  (NCMM) 

The  NCMM  agent  determines  the  most  effective  use  of  actors  to  counter  detected  threats.  Such  an  agent  enables 
co-ordinated  counter  measures  that  can  not  be  performed  manually  using  the  separate  systems.  Most  notably,  this 
involves  counter  measures  combining  jamming  and  chaff.  Furthermore,  the  manager  can  enhance  effectiveness 
by  combining  counter  measures  to  counter  more  than  one  threat  simultaneously  [Tempelman  1999], 
[Eertink  2000]. 

More  information  on  the  NCMM  can  be  found  in  the  second  part  of  this  paper,  which  describes  an  experiment  to 
study  the  effects  of  decision  support  in  the  military  cockpit  and  that  has  been  earned  out  with  the  NCMM. 

3.5  Display  Development  with  NADDES 

An  HMI  (Human  Machine  Interface)  is  designed  with  an  NLR  built  tool,  NADDES  (NLR  Avionics  Display 

Development  and  Evaluation  System)  for  increasing  pilot 
situation  awareness.  NADDES  is  a  development  environment 
that  has  specifically  been  developed  for  the  construction  of 
avionics  displays  and  as  such,  it  provides  a  number  of 
predefined  components. 

For  the  aforementioned  demonstrations,  separate  HMIs  have 
been  developed.  Since  all  demonstrations  use  the  same 
development  environment,  they  can  easily  be  integrated  to  form 
one  situation  awareness  picture  for  the  pilot.  The  display 
manager  agent  decides  which  information  to  display  on  the 
available  on-board  HMIs. 

An  example  of  a  NADDES  developed  HMI  for  the  NCMM  is 
given  in  figure  7. 


Figure  7  NCMM  display 
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PART  II:  A  THREAT  MANAGEMENT  DECISION  SUPPORT 
EXPERIMENT 


The  second  part  of  this  paper  describes  a  Crew  Assistant  and  Situation  Awareness  experiment  that  has  been 
earned  out  involving  operational  RNLAF  pilots  in  the  NSF  simulator  at  NLR,  Amsterdam.  We  will  first  give  an 
elaborated  overview  of  the  NCMM  tool  that  has  been  used  and  then  focus  on  the  details  of  the  experiment. 


The  current  military  theatre  becomes  ever  more  complex  and  demanding  for  the  pilot.  This  development  is 
potentially  threatening  to  the  situational  awareness  of  pilots  and  as  such,  to  their  safety.  Solutions  to  keep  track  of 
the  situation  are  sought,  amongst  others,  in  automated  tactical  decision  aids.  To  what  extend  will  pilots  benefit 
from  such  systems?  Will  mission  performance  increase?  What  are  the  disadvantages?  To  answer  these  questions 
an  experiment  was  designed,  which  will  be  described  in  this  part  of  the  paper.  The  experiment  can  be 
characterised  best  as  an  evaluation  by  fighter  pilots  of  a  threat  management  support  system. 


Figure  8 .  F-16  MLU  cockpit 


The  experiment  is  performed  in  the  NLR  National  Simulation  Facility 
(NSF),  see  figures  4  and  8.  This  simulator  is  configured  with  an  F-16  MLU 
cockpit.  Mission  profiles  and  controls  used  by  the  pilots  during  the 
experimental  runs  were  recorded.  Instrumentation  and  fittings  allowed 
variables  such  as  heartbeat  frequency,  eye  point-of-gaze,  pupil  size,  and 
eye  blink  rate  to  be  measured. 

Pilots  flew  through  scenarios  in  which  they  encountered  several  threats. 
The  taskload  between  the  scenarios  was  varied  by  the  amount  of  surface 
threats  that  were  implemented  in  the  scenarios.  In  half  of  the  test  runs  pilots 
had  to  deal  with  the  treats  manually,  while  in  the  other  half  an  automated 
system  initiated  appropriate  counter  measures  by  itself. 


Afterwards  pilot  performance,  subjective  and  objective  mental  workload  and  the  pilot  appreciation  of  the 
automated  threat  management  system  were  evaluated.  For  the  experiment,  we  used  the  NCMM,  mentioned  in 
section  3.4  and  further  extensively  described  in  chapter  4.  Chapter  5  will  present  the  experiment  set  up  and 
describe  the  results. 


4  NLR  Counter  Measure  Manager  (NCMM) 

To  countermeasure  SAMs,  the  pilot  has  several  options  available.  Generally,  the  best  course  of  action  is  to  avoid 
entering  SAM  rings,  but  this  cannot  always  be  avoided,  especially  when  pop-up  threats  are  encountered. 
Depending  on  the  type  of  threat  and  its  state,  the  pilot  can  jam  or  dispense  chaff  en  flares.  Usually,  these  counter 
measures  are  combined  with  a  manoeuvre. 

The  NCMM  agent  determines  the  most  effective  use  of  actors  to  counter  detected  threats.  Such  an  agent  enables 
co-ordinated  counter  measures  that  can  not  be  performed  manually  using  the  separate  systems  [Tempelman  1999], 
[Eertink  2000].  A  schematic  representation  of  the  manager’s  architecture  is  supplied  in  figure  9. 
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Figure  9.  The  NCMM  in  the  simulation  environment . 


The  MSDF  agent  delivers  environment  data,  most  notably  sensor  tracks  and  threat  information,  to  the  resource 
manager.  The  resource  manager  is  the  core  of  the  NCMM.  It  consists  of  two  rule-based  knowledge  bases:  the 
Reflex  Counter  Measure  (RCM)  Knowledge  Base  and  the  Counter  Measure  Technique  (CMT)  Knowledge  Base. 
The  first  associates  sensor  tracks  directly  to  single  counter  measures  (e.g.,  chaff,  flare,  a  jamming  technique,  a 
manoeuvre),  in  situations  where  immediate  action  is  required.  The  latter  connects  threats,  i.e.,  fused  sensor  tracks, 
under  certain  positional  conditions  to  a  series  of  counter  measures,  after  which  the  dynamically  planning  CMT 
Scheduler  fits  these  into  the  existing  counter  measure  schedule. 

An  example  of  the  represented  CMT  knowledge  is  given  in  table  1.  Rule  number  12  (highlighted  in  the  table),  for 
example,  represents  what  can  be  done  to  counter  a  threat  of  type  SAMI  in  track  (T)  mode,  which  has  threat  level 
(lethality)  4  on  a  scale  of  0  to  10.  The  SAMI  is  detected  by  two  sensors,  the  Radar  Warning  Receiver  (R)  and  the 
ALQ  (J).  With  an  expected  effectiveness  of  8  on  a  similar  scale,  it  can  be  countered  by  a  combination  of  Range- 
Gate  Pull-Off  (a  jamming  technique)  followed  by  dispensing  4  bundles  of  chaff  with  an  interval  of  0.2  seconds 
(4C0.2),  if  the  condition  on  the  threat’s  position  is  fulfilled. 

The  knowledge  in  the  resource  manager  is  highly  flexible.  It  can  be  adapted  easily  to  include  newly  constructed 
counter  measure  techniques  and  it  can  be  modified  to  reflect  the  availability  of  other  agents  in  the  aircraft  in 
which  the  NCMM  is  running. 

Various  operational  modes  are  available  in  the  NCMM.  These  determine  the  availability  of  assets  in  various 
circumstances.  For  example,  it  can  be  necessary  to  not  use  jamming  counter  measures,  as  the  enemy  can  detect 
these.  Selecting  the  operational  mode  ‘Run  Silent’  will  then  take  care  of  this.  Furthermore,  the  NCMM  agent 
supports  system  modes,  determining  the  amount  of  automation  of  the  system.  These  vary  from  manual,  in  which 
case  only  advises  are  given,  to  fully  automatic,  in  which  case  all  counter  measures  except  manoeuvres  are 
executed  automatically  and  in  time.  For  reasons  of  pilot  convenience,  manoeuvres  will  only  be  advised. 
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ID 

Threat  Mode 

TL 

Sensors 

Effectiveness 

GCMT 

1 

CON 

SAMI  S 
DITION:  None 

3 

R 

5 

OT 

#  Simple  example:  SAMI  search  radar  (low  TL)  detected  by  RWR  only;  Turn  to  evade  the  site. 

2  |  SAMI  S  3  R+J  8  N1S 

CONDITION:  None 

#  As  above,  only  now  the  site  is  also  detected  by  the  ALQ,  so  jamming  can  be  used  (Noise). 

12  SAMI  T  4  R+J  8  RGPO1C+4C0.2 

CONDITION:  Threat  at  (20o<F<60o)v(300o<F<340°) 

#  Sam  is  tracking  the  aircraft.  Combination  of  RGPO  and  chaff  is  effective,  but  only  at  certain 
azimuth  angles 

32  |  SAMI  L  9  R+J+MA  8  VGPOl 

CONDITION:  Time  Till  Intercepts 

#  see  number  34. 

34  |  SAMI  L  10  R+J+MA  5  6GtumInto 

CONDITION:  Time  Till  Intercepts 

#  Missile  is  launched,  detected  by  MAW.  Assuming  semi-active  missile,  VGPO  can  be 
attempted.  If  this  fails  (TTI<5s),  perform  last  ditch  manoeuvre 


Table  1.  Part  of  the  CMT  knowledge  base 

The  NCMM  was  evaluated  and  validated  in  a  semi-realistic  simulation  environment  called  ITEMS.  Various 
scenarios  were  flown  in  various  modes.  A  simple  scenario,  in  which  no  threats  overlap,  is  shown  in  figure  10.  The 
flexibility  of  the  manager  was  demonstrated  by  running  it  both  in  a  simulated  F-16  and  in  an  Orion  P3,  the  latter 
having  far  less  counter  measure  possibilities.  The  NCMM  demonstrated  improved  effectiveness  and  efficiency  of 
using  countermeasures  against  threats  in  a  number  of  multi-threat  scenarios,  compared  to  manually  threat 
countering.  Advantages  are  that  series  of  counter  measures  that  require  exact  timing  with  respect  to  each  other  can 
be  executed  automatically,  that  the  NCMM  can  combine  counter  measures  to  counter  multiple  threats,  and  that  the 
NCMM  can  take  the  effect  of  a  counter  measure  against  a  threat  to  other  threats  into  account  in  highly 
complicated  situations. 


Figure  10.  Example  scenario  for  NCMM  validation. 
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5  A  threat  management  support  experiment 

This  chapter  describes  the  experiment  set  up  and  the  results. 

5.1  Experiment  Description 

The  purpose  of  the  current  experiment  is  to  demonstrate  the  advantages  and  disadvantages  of  a  CA.  The 
experiment  will  show  whether  automation  about  threat  management  tasks  is  beneficial  to  the  mission  compared  to 
the  current  situation,  in  which  pilots  have  to  perform  these  tasks  without  such  support.  To  define  the  additional 
value  of  the  support,  the  mission  performance,  the  subjectively  and  objectively  measured  fatigue  and  workload  of 
the  pilots  has  beeen  investigated.  NCMM  served  as  the  threat  management  support  system  (the  CA). 

The  experiment  was  designed  to  answer  the  following  specific  questions: 

•  Does  decision  support  improve  pilot  performance  (fighter  performance,  mission  effectiveness),  especially  in 
situations  in  which  a  great  deal  of  the  pilot’s  attention  is  needed  for  other  tasks  (high  taskload  saturation)? 

•  Will  pilots  appreciate  the  tool  when  they  can  be  convinced  that  the  pilot  always  remains  in  control? 

•  Will  the  integrated  way  of  information  presentation,  that  is  part  of  the  CA  interface  in  the  current  experiment, 
aid  the  pilots  in  gaining  and  remaining  their  situational  awareness? 

•  Will  pilots  get  a  better  understanding  and  appreciation  of  the  tool  after  they  are  given  time  to  test  it 
thoroughly? 

•  Will  pilots  either  experience  a  workload  decrease,  a  performance  increase,  or  a  combination  of  both  as  a 
result  of  the  CA? 

•  The  moments  with  the  highest  workload  are  needed  here.  Heart  rate  during  those  moments  should  be 
compared  between  the  CA  on  and  the  CA  off. 

Nine  operational  male  F-16  pilots  from  the  Royal  Netherlands  Airforce  participated  each  for  one  day  in  the 
experiment.  The  right  Multi  Function  Display  (MFD)  was  reserved  entirely  for  NCMM.  Swapping  the  NCMM 
page,  or  removing  it  from  the  MFD,  was  not  possible.  The  pilot  was  informed  about  NCMM  system  mode  before 
the  run  started. 

During  half  of  the  runs  NCMM  operated  in  the  standby  mode.  In  this  mode  NCMM  provides  explanatory 
information  about  the  location  and  type  of  the  surface  threats  and  the  flight  path  only.  In  this  mode  NCMM  does 
provide  the  same  information  as  what  is  now  available  on  the  current  F-16  MLU  Multi  Function  Displays  in 
combination  with  the  Radar  Warning  Receiver  (RWR).  The  pilot  selected  and  executed  the  counter  measures 
against  the  threats  manually.  He  had  to  actively  dispense  chaff  and  flare  and  select  the  jamming  mode. 
Manoeuvres  had  to  be  executed  manually. 

During  the  other  half  of  the  runs  NCMM  operated  in  the  automatic  mode.  In  this  mode  NCMM  decides  when  to 
expend  chaff  and  flare  and  dispenses  them  automatically.  The  pilot  can  dispense  additional  chaff  and/or  flare 
manually  in  this  mode.  Also  the  appropriate  jamming  modes  are  selected  automatically.  In  addition,  NCMM  also 
advises  about  appropriate  manoeuvring.  However,  the  execution  of  the  manoeuvres  has  to  be  done  by  the  pilot. 

The  pilots’  task  was  to  fly  six  low  altitude  weapon  delivery  missions.  Each  run  had  the  same  mission  goals  but  the 
mission  profile  was  different  every  time.  Pilots  were  instructed  to  stay  to  the  assigned  flight  path  as  much  as 
possible.  They  were  also  asked  to  fly  low  altitude,  though  not  too  low  so  that  they  would  not  fly  below  radar 
coverage.  The  pilots  were  recommended  to  use  NCMM  advice  if  they  felt  that  it  was  tactically  sound,  but  to 
otherwise  execute  their  own  plan.  Each  scenario  consisted  of  a  short  flight  (duration  was  about  ten  to  fifteen 
minutes)  in  which  the  subjects  encountered  a  number  of  threats.  During  the  runs  several  threats  could  pop  up. 
Given  the  nature  of  NCMM  it  was  decided  to  use,  primarily,  surface  threats.  Pre-planned  threats  and  steer  points 
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were  chosen  such  that  the  missions  that  the  pilots  were  flying 
could  well  be  compared.  Pop-up  threats  always  combined  to  a  pre¬ 
planned  threat  in  such  a  way  that  a  logical  course  of  action  would 
follow.  An  example  of  a  mission  with  pre-planned  threats  is  given 
in  figure  11.  The  dashed  circles  represent  the  threats  with  a 
numbered  classification  (fictive).  The  line  is  the  route  to  be 
followed,  including  numbered  way  points.  Way  point  number  five 
is  the  target. 


Figure  11.  Example  mission  with  pre-planned  threats . 

5.2  Experiment  design 

In  one  condition  NCMM  provided  decision  support  (automatic  mode).  This  condition  was  compared  to  a 
condition  in  which  NCMM  provided  explanatory  information  about  the  threats  only  (standby  mode)  but  no  advice 
was  offered. 

The  effect  of  NCMM  was  investigated  both  under  high  task  load  and  low  task  load  conditions.  During  half  of  the 
runs  the  pilot  was  flying  in  a  low  workload  environment.  A  limited  amount  of  surface  threats  was  present  during 
those  runs.  During  the  other  half  of  the  runs  a  high  workload  environment  was  presented.  In  these  scenarios  more 
surface  threats  were  present  and  enemy  fighter  aircraft  could  be  encountered. 

The  benefits  of  NCMM  were  compared  between  the  two  NCMM  modes  and  the  two  levels  of  task  load.  The 
above  means  that  the  pilots  had  to  fly  scenarios  in  four  different  configurations. 

Since  pilots  had  to  fly  six  runs  each  and  because  it  was  undesired  that  they  transferred  experience  between 
scenarios,  three  different  mission  profiles  were  developed.  The  runs  were  assigned  over  these  profiles  and  the  four 
experimental  conditions.  This  approach  ensured  that  pilots  never  flew  the  same  run  twice.  It  also  ensured  that  they 
flew  three  runs  with  NCMM  in  standby  and  three  in  automatic  mode  and  it  finally  ensured  that  they  flew  three 
runs  in  a  high  and  three  in  a  low  workload  condition. 

5.3  Measurements  and  equipment 

During  the  missions,  all  information  that  was  exchanged  between  the  agents  and  between  the  agents  and  the  pilots 
was  logged.  Besides,  psychophysiological  measurements  were  taken  and  the  pilots  were  asked  to  fill  out 
questionnaires  at  several  moments  during  the  day. 

In  close  co-operation  with  operational  experts,  simulator  events  have  been  valued  in  terms  of  performance.  Based 
upon  the  hypotheses  a  selection  of  simulator  and  NCMM  variables  was  made,  in  order  to  be  recorded.  These 
comprise  amongst  others:  countermeasures  made  by  NCMM  (in  automatic  mode),  countermeasures  made  by  pilot 
(in  stand  by  mode),  mission  duration,  amount  of  time  spent  in  search  or  track  of  threats,  deviations  from  the  flight 
path,  and  weapon  delivery  events. 
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Heart  rate 

In  addition  to  the  simulator  logfiles,  other  objective  indicators  of  mental  workload,  namely  psychophysiological 
parameters,  were  recorded  by  the  Vitaport  1  system.  Those  comprise  the  electrocardiogram  (ECG)  and 
respiration.  ECG  was  measured  with  three  Ag/AgCl  electrodes.  One  was  attached  approximately  4  cm.  above  the 
jugular  notch  of  the  sternum,  one  at  the  apex  of  the  heart  over  the  ninth  rib,  and  the  ground  electrode  was  placed 
above  the  right  iliac  crest.  The  ECG  was  used  to  determine  the  Heart  Rate  (HR)  and  Heart  Rate  Variance  (HRV). 
Respiration  was  measured  using  a  pair  of  strain  gauge  transducers  around  the  chest  and  the  abdomen,  so  that  the 
influences  of  respiration  rate,  and  speech,  on  HR  V  could  be  filtered  out  later.  During  the  offline  data  analysis  HR 
artefact  correction  was  earned  out  according  to  a  procedure  described  by  [Mulder  1988]. 

Eye  Point-of-Gaze 

Eye  Point-of-Gaze  is  the  point  on  a  predefined  surface  were  an  imaginary  line  coming  straight  from  the  centre  of 
the  eye  crosses  that  surface  via  the  lens  of  the  eye.  As  such  this  is  the  central  point  in  the  pilot’s  field  of  vision. 
This  point  was  measured  by  means  of  an  EPOG-recorder  called  GazeTracker  [Mooij  1996].  The  duration  that  a 
pilot  looks  at  a  particular  area  of  interest,  is  called  a  “dwell”,  which  was  stored  in  a  computer  file.  In  addition  to 
the  dwell-times  the  scanning  pattern,  the  amounts  of  fixations,  the  pupil  diameter,  and  eye  the  blink  activity 
(which  permits  blink  rate,  duration,  and  other  measures  to  be  derived)  of  the  pilots’  left  eye  were  recorded  as 
indicators  of  fatigue  and  mental  and  visual  workload  [Harris  1986],  [Wilson  1987],  [Wilson  1993],  [Stem  1994]. 
The  scanning  behavior  was  considered  to  be  an  indicator  of  the  pilot’s  mental  state  and  focus  of  attention.  The 
commonly  accepted  assumption  was  made  that  if  a  pilot  looks  at  a  particular  area  of  interest  he  is  mentally 
processing  the  data  that  are  manifest  at  that  area. 

5.4  Results 

The  pre-  and  post  experiment  questionnaires  demonstrates  that  pilot  opinion,  regarding  a  number  of  NCMM 
related  items,  had  changed  after  using  the  NCMM  in  the  experiment. 


Comparison  before  -  after  questionnaire 


Figure  12 .  Responses  to  questionnaires  before  and  after  the  experiment  Answers  were  first  transferred  to 
Z-scores1.  After  that  the  average  over  all  pilots  was  calculated ,  and  plotted  in  this  figure . 


Z  scores  are  standardised  scores.  They  were  calculated  using  the  following  formula:  Z  score  =  (raw  score  -  mean)  /  standard  deviation. 
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In  figure  12,  the  pilot  ratings  before  and  after  the  experiment  are  displayed.  The  left  side  of  the  graph  shows  the 
answers  to  the  “system  design  issue”  questions.  The  right  half  shows  the  answers  to  the  “system  application  (use) 
issues”.  The  red  line  represents  the  average  of  all  pilot  responses  to  the  pre -experiment  questionnaire.  The  green 
line  represents  the  average  of  all  pilot  responses  to  the  post-experiment  questionnaire.  Distances  between  the  two 
lines  may  be  seen  as  an  indication  of  the  “change  of  mind”  of  the  pilots  after  using  NCMM  in  the  NSF  for  several 
hours. 

While  often  the  trends  of  both  lines  are  the  same,  there  are  differences  as  well.  Roughly  speaking  the  differences 
(distances  between  the  two  lines)  comprise  the  following  issues. 

With  respect  to  the  system  design  related  questions,  it  was  observed  that  the  pilots  were  more  convinced,  after 
using  NCMM,  that: 

•  It  performs  like  a  real  pilot. 

•  Integration  in  the  aircraft  may  be  adequate. 

•  It  does  not  show  too  much  irrelevant/distracting  details. 

•  NCMM  is  capable  of  taking  pilot  personal  preferences  into  account. 

•  NCMM  is  sufficiently  sensitive  to  specific  mission  demands. 

The  most  important  results  concerning  the  system  application  (use)  were  that  pilots,  after  using  NCMM,  were 
more  convinced  that: 

•  They  have  confidence  in  the  system  and  will  as  such  use  it. 

•  They  will  not  necessarily  (eventually)  loose  EW  related  skills  when  using  NCMM. 

•  They  will  not  pay  too  much  attention  to  NCMM. 

Situational  Awareness  (SA)  as  rated  by  the  pilots  themselves  and  by  an  observer  is  visualized  in  figure  13 

SA  Rating  (pilot  and  observer) 


high  low  WL  high  low 

Experimental  condition 


Figure  13 .  Pilot  and  observer  ratings  of  pilot  situational  awareness . 

Note  that  pilots  considered  their  SA  at  approximately  the  same  level  during  all  four  conditions,  but  that  the 
observer,  who  was  aware  of  everything  that  happened  in  the  scenarios  (even  those  things  that  pilots  could  never 
be  aware  off)  rated  the  pilot  SA  higher  during  the  missions  were  NCMM  was  running  in  automatic  mode. 

One  of  the  performance  indicators  that  was  monitored  was  the  jamming  mode  the  pilots  selected  versus  the 
jamming  mode  that  was  selected  by  NCMM.  While  watching  the  pilots  perform,  it  already  became  very  likely 
that  pilots  frequently  forgot  to  select  the  appropriate  jamming  mode,  while  NCMM  selected  the  right  jamming 
mode  as  soon  as  it  has  identified  a  new  (high  priority)  threat. 


5-18 


6  Conclusions  from  the  POWER  project 

With  the  increase  in  aircraft  speed  and  on-board  technological  developments  and  the  increasing  complexity  of 
military  environment  and  missions,  the  workload  of  the  fighter  pilot  is  rapidly  increasing.  Crew  assistant 
technology  is  aimed  at  reducing  the  pilot’s  workload  through  enhanced  situational  awareness. 

The  POWER  project  provided  a  large  scale  demonstrator  for  crew  assistant  technology.  An  architecture,  based  on 
multi-agent  technology  has  been  set  up,  where  different  examples  of  crew  assistants  have  been  integrated: 

•  A  profile  recognition  detects  profiles  in  the  assembled  picture  from  the  multi-sensor  data  fusion  process. 

•  A  manoeuvre  prediction  can  be  used  in  dogfight  situations  to  predict  and  anticipate  enemy  aircraft's 
manoeuvres. 

•  NCMM  is  the  NLR  Counter  Measure  Manager  to  assist  the  pilot  in  taking  counter  measures  against  ground 
threats. 

The  focus  of  the  project  has  been  decision  support  in  operational  fighter  aircraft  scenarios  through  the  integration 
of  AIP  and  HF  aspects  in  the  military  cockpit.  We  have  shown  the  integration  of  these  areas  by  examining 
different  AIP  techniques  and  their  integration  in  the  cockpit.  Display  design  has  been  carefully  taken  place  to  take 
the  HF  aspects  into  account.  A  step  has  been  made  in  the  quantitative  and  qualitative  effects  of  on-board  decision 
support  functions  in  an  experiment  where  operational  F-l  6  pilots  participated. 

We  examined  the  possibilities  for  real-time  and  especially  any-time  reasoning  in  on-board  application.  Any-time 
algorithms  provide  the  possibility  to  interrupt  the  reasoning  process  at  any  moment  thus  enabling  optimal 
processor  performance  and  use.  Promising  techniques  are  Bayesian  Belief  Networks  and  case  based  reasoning. 

Future  work  will  be  directed  to  the  maintenance  of  the  architecture  provided  and  to  the  further  integration  of 
agents,  both  functional  and  for  system  control  and  monitoring.  More  work  needs  to  be  earned  out  to  reasoning 
with  uncertainty.  The  environment  can  be  used  to  facility  more  experiments. 
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Challanges  for  the  future 

The  various  technical  items  described  during  the  Lecture  Series  were  covering  modem 
Information  Technology  applications  and  achievements  in  the  Situational  Awareness  and 
Decision  Aids  Systems. 

The  Authors  unanimously  agreed  in  suggesting  improvements  for  the  future  in  what  follows. 


Decision  Superiority 

Particular  emphasis  was  given  by  the  Authors  on  the  effort  in  implementing  the  information 
gathering,  processing,  managing  and  disseminating  Systems  to  improve  the  Operators  capabilities 
to  reach  decisions  better  and  faster  than  before. 

This  was  considered  as  the  capability  to  reach  the  Information  Superiority . 

The  ever  more  demanding  requirements  to  exploit  the  Sensors  resources  to  their  full  effectiveness 
and  the  Information  Technologies  to  their  best,  in  order  to  provide  to  the  Operators  the  most 
successful  decision-making  capabilities,  have  introduced  the  new  concept  of  Decision 
Superiority . 

This  because  the  Information  in  itself  could  be  imprecise,  sparse  or  massive  and  confusing, 
resulting  therefore  counterproductive  with  respect  to  the  time  critical  decision-  making. 

Thus,  there  is  the  need  to  use  a  higher  level  of  Information  Technologies  capabilities  enabling  the 
Operators  to  reach  the  best  possible  Decisions . 

To  do  so,  the  Decisions  will  have  to  be  reached  by  an  ever  more  integrated  mix  of  human  and 
machine  reasoning . 

This  represents  the  new  challenges  for  the  development  of  the  future  Decision  Aids  Systems. 


Beyond  Awareness 

The  result  of  the  Decision  Superiority  is  a  Beyond  Awareness  capability. 

This  means  that  the  Operators  perspective  on  Situation  Awareness  must  be  expanded  beyond  the 
simple  notion  of  a  common  operational  picture  or  consistent  tactical  picture. 

There  is  the  need  to  be  aware  of  what  the  information  means  and  what  to  do  with  it. 

In  summary  the  approach  can  be  considered  as  three  levels  that  build  upon  one  another: 

•  Acquiring,  disseminating  and  integrating  information 

•  Processing,  displaying  and  understanding  information 

•  Determining  how  to  act  on  information 


Paper  presented  at  the  RTO  SCI  Lecture  Series  on  <(  Tactical  Decision  Aids  and  Situational  Awareness”,  held  in  Amsterdam, 
The  Netherlands,  1-2  November  2001;  Sofia,  Bulgaria,  8-9  November  2001;  Madrid,  Spain,  12-13  November  2001; 
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6-2 


While  the  first  level  of  capability  involves  Sensors  and  other  external  data  to  provide  the  best 
knowledge  of  the  Battlespace ,  the  second  level  is  able  to  appraise  the  available  information  to 
identify  criticalities  and  to  correlate  observations  in  order  to  create  a  better  understanding  of  it. 
Finally  the  third  and  more  difficult  level  relates  to  support  the  decision  to  act. 

This  corresponds  to  the  decide  and  act  capability,  the  main  decision-maker  task. 

Thus,  the  challenge  for  the  future  is  to  complement  the  ability  to  collect,  process  and  disseminate 
information  with  a  more  accurate  capability  to  use  the  information. 

Areas  of  Information  Technology  improvements  are  seen  especially  in: 

•  Artificial  Intelligence 

•  Expert  Systems 

•  Faster-than-real-time  Simulation 

•  Intelligent  agents 


Battlespace  Dominance 

The  success  in  achieving  the  decide  and  act  capability  will  result  in  more  and  more  complete 
Battlespace  Dominance ,  in  line  with  the  always  increasing  Defense  needs. 


Human  and  Machine  Integration 

The  final  question  is  how  to  use  the  achieved  Decision  Superiority  and  Beyond  Awareness 
capabilities  in  the  most  effective  way. 

The  simplest  answer  could  be  to  extend  and  complement  the  human  operator  capabilities  with 
those  provided  by  the  machine  but  it  is  not  so  easy. 

It  is  necessary  to  avoid  the  “man-out-of-the-loop”  effect  which  could  lead  to  unexpected  and 
dangerous  uncontrollable  results;  in  the  other  hand  it  is  often  necessary  to  decide  and  react  so 
quickly  and  beyond  human  capabilities  to  counteract  a  situation. 

This  is  a  dilemma  that  is  necessary  to  solve  to  obtain  maximum  benefits  from  today  emerging 
technologies. 
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